If you’ve scaled a GPU node group on EKS and watched a pod sit in ContainerCreating for three or four minutes while your CUDA/PyTorch image pulls, you’ve probably assumed it’s a bandwidth problem and reached for a bigger instance network tier. AWS’s EKS team published the actual root cause on August 10, 2026, and it’s not bandwidth at all — it’s serialization. Two containerd fixes, now default in EKS Auto Mode, cut that wait from minutes to seconds without touching your network config.

Modern GPU instances (p5, p5e, trn2 families) ship with 100-400 Gbps network interfaces. A 20-30GB CUDA/PyTorch/vLLM image should, on paper, pull in single-digit seconds at that bandwidth. In practice, teams routinely saw 2-5 minute pulls on the exact same instances. The instinct is to blame the registry, the VPC endpoint, or the NIC — but the actual bottleneck was how containerd processes an image pull, not how fast bytes move.

Root Cause: Sequential, Not Parallel

The default OCI image pull path in older containerd does two things that don’t scale with image size, regardless of link speed:

  1. Layers download as a single sequential HTTP GET per layer, with the full layer buffered before the next step begins — no range requests, no chunking. A single 8GB CUDA base layer is one long-lived HTTP connection, subject to TCP slow-start and connection-level throughput ceilings that a 400Gbps NIC never gets a chance to saturate.
  2. Layer unpacking (untar + checksum verification) happens one layer at a time, in sequence. Even after a layer finishes downloading, containerd doesn’t start unpacking layer N+1 until layer N is fully extracted to disk. For a multi-gigabyte image with 10+ layers, this serializes what should be an embarrassingly parallel operation.

Neither of these is a bandwidth problem. You can have an idle 400Gbps link and still be bottlenecked by TCP slow-start on a single stream and single-threaded tar extraction.

The Fix: Two Upstreamed containerd Changes

containerd 2.1 introduced chunked, range-request-based downloads that stream layer content straight to disk instead of buffering a full layer in memory before writing:

# conceptually, what changed — one big sequential GET per layer:
curl -o layer.tar.gz https://registry/v2/blobs/sha256:abc123...

# becomes concurrent ranged requests streamed to disk:
curl -r 0-99999999    -o layer.tar.gz.part1 https://registry/... &
curl -r 100000000-199999999 -o layer.tar.gz.part2 https://registry/... &
curl -r 200000000-299999999 -o layer.tar.gz.part3 https://registry/... &
wait

This lets a single large layer saturate multiple TCP streams concurrently instead of living or dying by one connection’s throughput ceiling — exactly the kind of thing that matters once you’re on a 100+ Gbps NIC where a single TCP stream realistically won’t hit line rate anyway.

containerd 2.2 tackled the second bottleneck: concurrent layer unpacking instead of strictly sequential. Layers with no dependency ordering conflicts (most layers in a typical CUDA base image) now unpack in parallel as soon as their bytes are on disk, instead of waiting for every prior layer to finish extraction first.

Both are now the default behavior in EKS Auto Mode, and the EKS team published specific config guidance for both AL2023 and Bottlerocket AMIs if you’re managing your own node groups outside Auto Mode:

# containerd config.toml — relevant knobs (AL2023 self-managed nodes)
[plugins."io.containerd.grpc.v1.cri".containerd]
  discard_unpacked_layers = false

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

# enable concurrent pull/unpack (containerd >= 2.1/2.2)
[plugins."io.containerd.snapshotter.v1.overlayfs"]
  sync_remove = false

What This Actually Means for Node Scale-Up Latency

If you run inference workloads on EKS — autoscaling a GPU node group in response to a traffic spike, or spinning up ephemeral nodes for batch training jobs — image pull time is directly on your scale-up critical path. A 3-4 minute pull on top of node bootstrap and kubelet readiness easily pushes cold-start latency for a new GPU node past 5-6 minutes. Cutting the pull itself to tens of seconds is the difference between “autoscaling reacts to load within a reasonable SLA” and “your on-call gets paged because the scale-up didn’t land before the traffic spike passed.”

Concretely, for a team running vLLM or Triton inference servers on EKS: if you’re on self-managed node groups (not Auto Mode), check your containerd version and config before assuming your slow pulls are a network or ECR issue. This is a five-minute config check that can save you from over-provisioning warm standby nodes purely to work around pull latency.

# quick check on a self-managed EKS node
containerd --version
# if < 2.1, the range-request download optimization isn't available
# if < 2.2, layer unpacking is still sequential

# verify current pull behavior via containerd's own metrics
ctr --namespace k8s.io images pull --local  registry/image:tag

The Named Roadmap Item Worth Watching

The EKS team’s writeup also flags two upcoming changes worth tracking: rapidgzip (parallel-decompressible gzip decoding, since standard gzip decompression is itself single-threaded and can bottleneck even after download and unpack are parallelized) and BLAKE3 tree-hashing for layer verification (replacing SHA-256’s sequential hash chain with a tree structure that can be verified in parallel). Neither has shipped as of this writing, but both attack the same class of problem — steps in the pull pipeline that were designed for small images and never revisited as GPU/ML images grew into the tens of gigabytes.

The Lesson for Platform Teams

This is a good case study in a pattern I keep running into: when infrastructure that was designed for a different scale starts underperforming, the instinct is almost always “throw more bandwidth/compute at it,” and that instinct is usually wrong. Container image pulling was designed in an era of few-hundred-MB application images; it was never re-architected for 20-30GB ML images, and no amount of NIC upgrade fixes a fundamentally sequential pipeline. The actual fix required someone to profile where the wall-clock time was actually going — TCP slow-start on single streams, single-threaded tar extraction — rather than assuming the bottleneck was where it “should” be.

If you’re running GPU workloads on Kubernetes anywhere (EKS or otherwise) and haven’t checked your containerd version against this, it’s worth the five minutes. It’s one of those fixes that costs nothing and just makes your scale-up faster.

Thuận Lương is a Technical Lead with 15+ years in .NET, cloud architecture, and AI systems. He writes about real-world lessons from building production systems.

Export for reading

Comments