Image Volumes Went GA in Kubernetes 1.36 - Should You Ship Your Models as OCI Images?

Every model server I have run on Kubernetes starts the same way: an init container downloads several gigabytes of weights from somewhere, every time a pod starts, on every node. The alternatives were a PVC you populate and pin, or weights baked into a 10 GB runtime image you rebuild for every model bump.

Image volumes went GA in Kubernetes 1.36. You push the weights as an OCI image, reference it from the pod spec, and the kubelet mounts it read-only. Same registry, same pull secrets, same digests as your containers. The question I wanted answered is simple: is it faster, and when should a platform team switch?

Provenance: every number below is from one kind cluster (kindest/node:v1.37.0, containerd 2.3.4) on an Apple Silicon laptop, a local registry on the kind network, vLLM CPU v0.33.0 and Qwen2.5-Instruct 0.5B (1 GB) and 1.5B (3 GB). Absolute numbers are laptop numbers. The ratios are the result.

TL;DR#

• Warm node: the image volume costs 0 seconds of delivery, the same as a pre-populated PVC, without a populate Job or a storage class.
• Cold node: pulling the weights image took 5.5s for 1 GB and 17.8s for 3 GB. A plain HTTP download of the same files from the same network took 2.2s and 4.7s. Cold image volumes are 2.5 to 4 times slower.
• Internet: downloading from the Hugging Face Hub took 85s and 259s. Anything local beats it by an order of magnitude.
• The runtime dominates small models: vLLM’s engine init took 25 to 40 seconds on every run, longer than every local delivery method.
• The volume is the image root. Without subPath, vLLM cannot find your model.
• The pull is invisible in pod conditions. It happens after PodReadyToStartContainers and shows up only in a Pulled event.

What image volumes are#

A pod volume of type image points at an OCI image reference. The kubelet pulls it through the container runtime and mounts its filesystem read-only into the containers that ask for it. The KEP-4639 history is alpha in 1.31, beta in 1.34, stable in 1.36. On the runtime side, containerd added support in v2.1.0 and subPath mounts in v2.2.0.

KServe already uses it: oci+native:// storage URIs became image volume mounts in v0.20.0. You do not need KServe to use it, and this post does not.

What it is NOT#

• Not lazy loading. The whole image is pulled and unpacked before any container starts.
• Not a model registry. It is a registry for bytes. Versioning is your tag and digest discipline.
• Not a cache you control. The node’s containerd store is the cache, with the kubelet’s image GC deciding when it goes away.

The options I compared#

Bottom line: image volumes trade a slower first start per node for zero cost on every start after that, with the same supply chain as your containers.

The demo#

flowchart TB
    subgraph Host["Laptop (Docker Desktop)"]
        REG["kind-registry<br/>weights + vLLM images"]
        HUB["huggingface.co"]
    end
    subgraph Kind["kind: imgvol-demo (K8s 1.37.0, containerd 2.3.4)"]
        subgraph CP["control-plane"]
            FS["fileserver (nginx)<br/>same weights over HTTP"]
        end
        subgraph W["worker"]
            IV["vLLM pod<br/>image volume"]
            DL["vLLM pod<br/>init download"]
            PVC["vLLM pod<br/>PVC"]
        end
    end
    REG -->|pull| IV
    FS -->|HTTP| DL
    HUB -->|HTTPS| DL
    FS -->|populate Job| PVC

The weights image has nothing in it except the files:

# data-only image: no shell, no runtime, just the weights under /models
FROM scratch
COPY model/ /models/

The in-cluster mirror is an nginx pod that serves the same weights image through its own image volume. That keeps the download and the pull on the same bytes and the same Docker network, so the comparison is about the delivery path, not the source.

Gotcha 1: the volume is the image root#

The first pod mounted the image at /models and pointed vLLM at it. It crashed:

pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
  Value error, Invalid repository ID or local directory specified: '/models'.
   - For Hugging Face models: ensure the presence of a 'config.json'.

The image volume mounts the image’s root filesystem. The Dockerfile put the files under /models, so inside the pod they were at /models/models/config.json. The fix is subPath, which needs Kubernetes 1.33+ and containerd 2.2+:

      volumeMounts:
        - name: weights
          mountPath: /models
          # the volume is the image root, files live under /models inside it
          subPath: models
          readOnly: true
  volumes:
    - name: weights
      image:
        reference: localhost:5001/weights/qwen2.5-1.5b:v1
        pullPolicy: IfNotPresent

KServe’s oci+native:// mounts with subPath: models for the same reason. Build your weights images with a fixed directory and use subPath everywhere, or copy to / and skip it. Pick one convention for the whole platform.

Set pullPolicy explicitly. The docs default it to Always for :latest, and Always throws the node cache away on every start, which is the whole point of the feature.

Result: one line of YAML, and a crash message that says nothing about volumes.

Gotcha 2: the pull hides between the conditions#

I wrote a small timeline.sh that prints each pod condition relative to creation, the kubelet Pulled events and vLLM’s own timings. A cold 1.5B run:

== qwen-imagevolume-cold-15b: seconds since creation (API timestamps, 1s precision)
  scheduled          0
  sandbox_ready      0
  initialized        0
  container_started  20
  ready              85
== image pulls (kubelet events)
  localhost:5001/weights/qwen2.5-1.5b:v1       19.612s
  localhost:5001/vllm/vllm-openai-cpu:v0.33.0  cached
== runtime
  Loading weights took 0.66 seconds
  init engine (profile, create kv cache, warmup model) took 40.72 s

PodReadyToStartContainers and Initialized are both true at second 0. The container starts at second 20. The 19.6 seconds in between are the weights pull, and no condition covers them. On the first run, with the vLLM image also cold, the events show the order: weights image first, runtime image second, then the container.

Result: if your startup dashboards are built on pod conditions, image volume pulls look like a slow container runtime. Alert on the Pulled event duration or the kubelet’s image pull metrics instead.

The numbers#

Delivery only#

To measure delivery without vLLM noise, delivery.sh starts pods that only touch the weights: a container that lists the image volume, or one that downloads the files with curl and reports the time. Cold means I removed the weights images from the worker with crictl rmi before the round. Three rounds, two for the Hub:

Medians, range in brackets. The populate Job for the PVC took 9s and 15s, once.

The cold image volume is the most stable row in the table and the slowest local one. The KServe benchmark saw the same on x86 with 2 to 140 GB artifacts, about 2x slower than S3 at 140 GB, and explains it: containerd writes the layer blob and then unpacks it into a snapshot, two passes over the disk, where a download streams once. The layers are also gzip-compressed by default. The Pulled event reports 776 MB for 954 MB of files, so the node decompresses as well. I did not isolate which of these costs the most.

Pod to Ready#

The same methods with vLLM serving the 1.5B model, one run each:

PVC and warm image volume deliver nothing and still differ by 8 seconds. That is the run-to-run noise of vLLM’s engine init, 25 to 40 seconds of profiling, KV cache allocation and warmup on CPU. Weight loading itself took under 2.5 seconds in every run. For small models on CPU, the runtime is the cold start. Delivery grows with the model and the runtime does not: in the KServe benchmark the cold image volume path took 443 seconds at 14 GB, while the warm path stayed at 11 seconds from 2 GB to 140 GB.

Things that broke and were not image volumes#

The last one is a warning for real clusters too. Weights images are large, and the node image GC now decides how long your warm cache lives. Size the node disk for the models you expect to keep warm, not only for container images.

When to ship models as OCI images#

Pros and Cons#

Hands-On Demo Repository#

Everything above is in srekubecraft-demo/image-volumes:

task up                         # kind + local registry, both models, mirrors, PVCs
task run M=imagevolume          # one vLLM pod to Ready with the phase timeline
task run M=pvc SIZE=1.5b        # M=imagevolume|pvc|lan|hf
task delivery R=1 HF=1          # delivery-only round, appends to results/delivery.tsv
task cold                       # prints the crictl rmi for the next cold round

The cluster needs about 25 GB of disk and 16 GB of memory for the 1.5B runs. task cold prints the eviction command instead of running it, because removing images from a node is a decision you should see.

Conclusion#

Image volumes are the first native way to ship model weights through the infrastructure you already run for containers. On a warm node they cost nothing, and on a cold node they cost more than a plain download. Whether that trade wins depends on how often a model returns to the same node.

For most inference platforms it does: rollouts, restarts and scale-ups land on nodes that served the model before. If you download from the Hub at every start today, switch to a registry next to your nodes: in my runs even the cold pull was 14 times faster than the Hub. If you already serve weights from storage next to the nodes, switch for the supply chain, not for speed.

Before you do, set subPath and pullPolicy explicitly, add the Pulled event duration to your startup dashboards, and size node disks for the models you want to keep warm.

If you found this useful, you might also enjoy my related posts on AI workloads on Kubernetes:

• KServe - Production ML Serving on Kubernetes, from sklearn to LLMs
• llm-d - Kubernetes-Native Distributed LLM Inference at Scale
• Immutable Kubernetes Nodes as OCI Images - A Close Look at bootc