• AWS EKS costs can escalate due to massive, parallel workloads in life sciences/drug development (e.g., genomic sequencing, molecular modeling).
• Default Kubernetes scheduler uses leastAllocated strategy, spreading pods across many nodes for fairness/high availability.
• leastAllocated strategy causes many partially utilized nodes, preventing autoscalers from scaling down idle nodes, increasing costs.
• mostAllocated scheduling strategy "packs" pods onto fewer nodes, maximizing utilization and enabling autoscalers like Karpenter to remove idle nodes.
• Switching to mostAllocated can reduce runtime costs significantly (e.g., ~10% in UAT, 43% in PROD environments).
• Custom scheduler deployment on AWS EKS requires creating a service account, ClusterRoleBindings, RoleBinding, a ConfigMap with the mostAllocated scoring strategy, and a deployment with a matching Kubernetes version container image.
• Resource weights can prioritize packing of expensive resources (e.g., high weight on GPUs for ML workloads).
• Testing in non-production environments is recommended before full rollout.
• Implementing mostAllocated scheduling can dramatically optimize costs by enabling cluster autoscalers to shut down unused nodes.

See the summary below this highlight

• Introduction

  • Containers are widely used for scalable applications but face challenges with startup times for large images (e.g., AI/ML workloads).
  • Pulling large images from Amazon Elastic Container Registry (ECR) can take several minutes, impacting performance.
  • Bottlerocket, an AWS open-source Linux OS optimized for containers, offers a solution to reduce container startup time.

• Solution Overview

  • Bottlerocket's data volume feature allows prefetching container images locally, eliminating the need for downloading during startup.
  • Prefetching is achieved by creating an Amazon Elastic Block Store (EBS) snapshot of Bottlerocket's data volume and mapping it to new Amazon EKS nodes.
  • Steps to implement:
  • Spin up an Amazon EC2 instance with Bottlerocket AMI.
  • Pull application images from the repository.
  • Create an EBS snapshot of the data volume.
  • Map the snapshot to Amazon EKS node groups.

• Benefits of Bottlerocket

  • It separates OS and container data volumes, ensuring consistency and security during updates.
  • Prefetched images significantly reduce startup times for large containers.

• Implementation Walkthrough

  • Step 1: Build EBS Snapshot
    • Automate snapshot creation using a script.
    • Prefetch images like Jupyter-PyTorch and Kubernetes pause containers.
    • Export the snapshot ID for use in node group configuration.
  • Step 2: Setup Amazon EKS Cluster
    • Create two node groups:
    • no-prefetch-mng: Without prefetched images.
    • prefetch-mng: With prefetched images mapped via EBS snapshot.
  • Step 3: Deploy Pods
    • Test deployment on both node groups.
    • Prefetched nodes start pods in just 3 seconds, compared to 49 seconds without prefetching.

• Results

  • Prefetching reduced container startup time from 49 seconds to 3 seconds, improving efficiency and user experience.

• Further Enhancements

  • Use Karpenter for automated scaling with Bottlerocket nodes.
  • Automate snapshot creation in CI pipelines using GitHub Actions.

• Cleaning Up

  • Delete AWS resources (EKS cluster, Cloud9 environment, EBS snapshots) to avoid charges after testing.

• Conclusion

  • Bottlerocket's data volume prefetching dramatically enhances container startup performance for large workloads on Amazon EKS.

In this case, IaC is favored over using EKS directly or manually deploying on EC2

Why use EKS

Cost of EKS

Benefits of using Kubernetes on AWS: - scalability - cost efficiency - high availability

Karpenter

Key limitation of CAS