llm-d - Kubernetes-Native Distributed LLM Inference at Scale

Three months ago I wrote about KServe and how the InferenceService CRD had become the closest thing cloud-native has to a standard for putting a trained model behind an API. That post ended on a deliberate cliffhanger: KServe gives you a great single-model serving primitive, but it does not solve GPU sharing, fractional scheduling, or how you load-balance inference traffic across many replicas of a large model. I pointed at Volcano and Kueue and moved on.

This post is the other half of that story. Once your models get big enough and your traffic high enough, the interesting problem is no longer “how do I serve one model” - it is “how do I route, disaggregate, and cache inference across a fleet of GPUs so tail latency stays flat under load.” A plain Kubernetes Service in front of eight vLLM pods is inference-blind: it round-robins requests as if every token cost the same. It does not. That gap is exactly what llm-d fills.

llm-d joined the CNCF as a Sandbox project at KubeCon EU 2026, jointly donated by IBM Research, Red Hat, and Google Cloud, with founding support from NVIDIA, AMD, CoreWeave, Hugging Face, Intel, Lambda, and Mistral AI. It is a Kubernetes-native distributed inference framework built on top of vLLM, the Gateway API Inference Extension, and LeaderWorkerSet. This post walks through what it is, why plain Kubernetes load balancing falls short for LLMs, and how to run the whole orchestration layer on a Kind cluster on your laptop - no GPU required - using the vLLM simulator and Flux GitOps. The full demo lives in srekubecraft-demo/llm-d/.

Who Should Read This?#

This post is for:

• Platform Engineers building an internal LLM inference platform who have outgrown “one vLLM Deployment behind a Service”
• SREs who need to operate LLM serving with predictable tail latency, autoscaling, and GitOps like any other production workload
• ML Platform Teams already running KServe or raw vLLM who are hitting cache-thrash and uneven GPU utilization under real traffic
• Engineers evaluating the 2026 inference stack - vLLM, KServe, llm-d, KAITO, Kueue - who want to know where llm-d fits
• Anyone who saw the CNCF Sandbox announcement and wants to understand what llm-d actually does before committing GPUs to it

TL;DR#

Problem: Serving a large model at scale means running many replicas across many GPUs. A standard Kubernetes Service load-balances them with round-robin or least-connections - blind to prompt cache locality, KV-cache pressure, and the fact that the prefill and decode phases of a request have completely different resource profiles. The result is cache thrash, uneven GPU utilization, and unpredictable p99 latency.

Solution: llm-d is a Kubernetes-native distributed inference framework. It replaces the dumb load balancer with an inference-aware scheduler (built on the Gateway API Inference Extension), disaggregates prefill and decode into independently scalable pods, and adds hierarchical KV-cache offload across GPU, CPU, and storage tiers. The engine underneath is vLLM.

Result: Requests are routed to the replica most likely to already hold the prompt’s prefix in cache, prefill and decode scale independently, and GPU memory is used far more efficiently. In this post I run the entire orchestration layer - scheduler, gateway, endpoint picker, model servers - on a Kind cluster with zero GPUs by swapping real vLLM for the llm-d-inference-sim simulator, all delivered via Flux. Full demo repo.

The LLM Load-Balancing Problem#

Put eight vLLM replicas behind a Kubernetes Service and send it production traffic. It works. Until you look at the metrics.

Why round-robin is wrong for LLMs:

• Prompt cache locality is invisible. vLLM keeps a prefix cache: if two requests share a long system prompt, the second one can reuse the first’s KV-cache and skip recomputing it. A round-robin Service scatters those two requests to different pods, so the second one recomputes everything from scratch. On a 4k-token shared prefix, that is the difference between a 40ms and a 2000ms time-to-first-token.
• Prefill and decode are different workloads. The prefill phase processes the whole prompt in one compute-heavy forward pass - it is bound by GPU FLOPs. The decode phase generates tokens one at a time - it is bound by memory bandwidth and KV-cache size. Cramming both onto the same pod means a big prefill request head-of-line-blocks every decode stream sharing that GPU.
• KV-cache pressure is per-pod and dynamic. A pod running twelve long-context conversations is near its KV-cache limit and will start evicting or queuing. A round-robin balancer keeps sending it new requests anyway, because it only counts connections, not GPU memory.
• Queue depth varies wildly. One in-flight request generating 2000 tokens holds a slot for 30 seconds. The balancer sees “one connection” and treats that pod as lightly loaded.

None of this is visible to a standard L4/L7 load balancer, because none of it is expressed in HTTP. The routing decision needs to see inference state - cache contents, KV-cache utilization, queue depth - which lives inside the model server. That is the whole premise of llm-d.

What is llm-d?#

llm-d is a Kubernetes-native, high-performance distributed LLM inference framework. It is not a new inference engine - it uses vLLM (and can front SGLang and TensorRT-LLM) as the engine. What llm-d adds is the orchestration around the engine: inference-aware routing, prefill/decode disaggregation, and tiered KV-cache management, all expressed as standard Kubernetes primitives.

It was accepted into the CNCF Sandbox in March 2026. The latest release as of this writing is v0.8.1 (June 2026). The project ships “well-lit paths” - documented, tested, benchmarked deployment recipes - rather than a single monolithic install.

llm-d rests on three pillars.

Pillar 1: Inference-aware routing (Gateway API Inference Extension)#

The routing brain is an Endpoint Picker (EPP) built on the Gateway API Inference Extension (GAIE), a Kubernetes SIG project that extends the Gateway API with an InferencePool resource and a pluggable scheduler. Instead of round-robin, the EPP scores every candidate endpoint using a chain of scorers:

• prefix-cache scorer - estimates how much of the incoming prompt’s prefix each pod already holds in cache, and prefers the best hit
• no-hit LRU scorer - for cold requests with zero cache hits, spreads them evenly to balance the prefill load
• kv-cache utilization scorer - penalizes pods near their KV-cache limit
• queue-depth scorer - penalizes pods with deep request queues

The result is a routing decision that maximizes cache reuse while avoiding hotspots - the opposite of round-robin.

Pillar 2: Prefill/decode disaggregation#

llm-d can split the two phases of inference into separate pods that scale independently. Prefill pods run on compute-optimized GPUs and handle the heavy prompt-processing pass; decode pods run on bandwidth-optimized GPUs and stream tokens. The KV-cache computed during prefill is transferred to the decode pod over a fast connector (NIXL/NCCL). This lets you scale prefill and decode capacity to match your actual prompt-length-to-output-length ratio instead of over-provisioning both. Multi-node model replicas are orchestrated with LeaderWorkerSet (LWS).

Pillar 3: Hierarchical KV-cache offload#

KV-cache is the memory of a conversation, and GPU HBM is scarce. llm-d integrates LMCache to offload KV-cache across a tiered hierarchy - GPU HBM, then CPU RAM, then local disk or shared storage - so a warm conversation can be resumed without recomputation even after it has been evicted from the GPU. This is what makes long, multi-turn sessions affordable.

flowchart TB
    Client["Client / OpenAI SDK"] --> GW["Inference Gateway<br/>(Gateway API + GAIE)"]

    subgraph LLMD["llm-d control + data plane"]
        GW --> EPP["Endpoint Picker (EPP)<br/>prefix · kv-util · queue scorers"]
        EPP -->|routing decision| POOL

        subgraph POOL["InferencePool"]
            direction LR
            subgraph Prefill["Prefill pods (compute-bound)"]
                P1["vLLM prefill"]
                P2["vLLM prefill"]
            end
            subgraph Decode["Decode pods (bandwidth-bound)"]
                D1["vLLM decode"]
                D2["vLLM decode"]
            end
            Prefill -->|KV transfer<br/>NIXL/NCCL| Decode
        end

        Decode <--> KV["LMCache tiers<br/>GPU HBM → CPU → disk"]
    end

    style LLMD fill:#0d1117,stroke:#30363d,color:#c9d1d9
    style POOL fill:#161b22,stroke:#30363d,color:#c9d1d9

What llm-d is NOT#

• Not an inference engine. It orchestrates vLLM/SGLang/TensorRT-LLM; it does not replace them. If you want raw single-pod serving, run vLLM directly.
• Not a model-serving CRD like KServe. There is no InferenceService abstraction that wraps predictive ML, ONNX, and sklearn. llm-d is laser-focused on large generative models at scale. See the comparison below.
• Not a training or fine-tuning platform. That is Kubeflow, Ray, or your own pipelines. llm-d is inference-only.
• Not a GPU scheduler. It assumes accelerators are available and schedulable. Pair it with the NVIDIA GPU Operator, Kueue, or Volcano for the actual GPU plumbing.
• Not magic on one GPU. If you serve a small model with low QPS on a single replica, llm-d’s routing and disaggregation buy you nothing. Its value shows up at multi-replica, multi-node scale.

llm-d vs the alternatives#

Bottom line: KServe and llm-d are not competitors - they solve different layers, and the convergence already happened. KServe shipped LLMInferenceService in v0.16, and the docs are explicit that it is “built on the foundation of llm-d”. KServe creates the InferencePool and runs the llm-d Endpoint Picker on your behalf, so you get the cache-aware routing from this post through a KServe CRD.

That makes the stack a clean three-layer split rather than a choice:

So the mental model is not “KServe or llm-d”. It is: classic InferenceService for the sklearn/ONNX/XGBoost fleet, LLMInferenceService for the handful of large generative models that dominate your GPU bill, both on one control plane. Running llm-d standalone - as this demo does - is still the right way to understand the routing layer, and the right choice if you are not on KServe.

The GPU-free demo: llm-d on Kind with the vLLM simulator#

Here is the problem with writing a hands-on llm-d post: the whole point of the project is distributing inference across GPUs, and GPUs are exactly what a laptop does not have. Renting an 8xH100 node to write a blog demo is neither cheap nor reproducible for readers.

The unlock is the llm-d-inference-sim - a lightweight, OpenAI-compatible simulator that mimics vLLM’s behavior (token streaming, latency patterns, prefix-cache hits, KV-cache metrics, even prefill/decode disaggregation) without a GPU or a real model. It is the same tool the llm-d maintainers use to test the scheduler and routing logic at scale in CI. That means we can run the entire llm-d orchestration layer - gateway, endpoint picker, scorers, InferencePool, disaggregation topology - on a Kind cluster and observe the routing decisions, while the “model servers” are featherweight sim pods.

What this demo proves: the control plane, the inference-aware routing, and the disaggregation topology. What it does not prove: real tokens/sec on real hardware. For that you need GPUs, and I cover the production path below.

Architecture of the demo#

flowchart TB
    subgraph Kind["Kind cluster (laptop, no GPU)"]
        GAIE["GAIE CRDs v1.5.0<br/>(applied server-side)"]

        subgraph Flux["Flux HelmRelease"]
            EPP["llm-d router v0.9.0<br/>Endpoint Picker + Envoy<br/>+ InferencePool"]
        end

        SIM["llm-d-inference-sim x4<br/>(fake vLLM, OpenAI-compatible)"]

        GAIE --> EPP
        EPP -->|"scorer chain picks endpoint"| SIM
        EPP -.->|"InferencePool selects app=qwen-sim"| SIM
    end

    Dev["curl / OpenAI SDK"] -->|"/v1/completions"| EPP

    style Kind fill:#0d1117,stroke:#30363d,color:#c9d1d9
    style Flux fill:#161b22,stroke:#30363d,color:#c9d1d9

Prerequisites#

# Tooling (Apple Silicon or Linux)
kind version        # v0.23+
kubectl version     # v1.30+
flux --version      # v2.3+
helm version        # v3.12+

Step 1: Bootstrap the cluster (Kind + Cilium + Flux)#

The demo follows the same imperative-bootstrap-then-GitOps pattern as the rest of the srekubecraft-demo monorepo. A Taskfile drives everything, so the whole run is one command:

git clone https://github.com/nicknikolakakis/srekubecraft-demo
cd srekubecraft-demo/llm-d
task setup    # kind + cilium + flux -> GAIE CRDs -> router -> sim -> ready

Under the hood, bootstrap creates a three-node Kind cluster (control-plane + a system worker for the router + an inference worker for the sim pods), installs Cilium as the CNI, and installs Flux.

Step 2: Gateway API Inference Extension CRDs#

The router’s InferencePool lives in the inference.networking.k8s.io API group, which comes from the Gateway API Inference Extension. The demo pins it to v1.5.0 to match llm-d v0.8.x’s guides/env.sh, applied server-side because CRD manifests are large:

# kubernetes/gaie/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/v1-manifests.yaml

Step 3: The simulated model servers#

Instead of a vllm/vllm-openai container demanding nvidia.com/gpu: 1, the model-server Deployment runs the simulator. Note the total absence of GPU resource requests - and that the app: qwen-sim label is what the router’s InferencePool selects on:

# kubernetes/modelserver/sim-deployment.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-sim
  namespace: llm-d
  labels:
    app: qwen-sim
spec:
  replicas: 4
  selector:
    matchLabels:
      app: qwen-sim
  template:
    metadata:
      labels:
        app: qwen-sim
    spec:
      nodeSelector:
        workload: inference          # land on the simulated GPU pool
      containers:
        - name: vllm-sim
          image: "ghcr.io/llm-d/llm-d-inference-sim:v0.10.0"
          args:
            - "--model=Qwen/Qwen3-0.6B"
            - "--port=8000"
            - "--mode=random"          # random | echo
            - "--max-num-seqs=16"
          ports:
            - name: http
              containerPort: 8000
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "250m"
              memory: "256Mi"

Four “model servers” fit comfortably on a laptop because each one is a Go binary pretending to be a 0.6B model, not the model itself.

Step 4: The router - Endpoint Picker, proxy, and InferencePool#

You do not hand-write the InferencePool. The llm-d router chart (oci://ghcr.io/llm-d/charts/llm-d-router-standalone, v0.9.0) provisions the Endpoint Picker (EPP), a bundled Envoy proxy, and the InferencePool - all from one set of values. Standalone mode needs no external Gateway or LoadBalancer, which is exactly why it runs on Kind. The HelmRelease (owned by Flux) points the pool at the sim pods and wires up the scorer chain:

# kubernetes/flux/router-release.yaml (values excerpt)
values:
  router:
    modelServers:
      matchLabels:
        app: qwen-sim          # selects the sim pods into the InferencePool
      targetPorts:
        - number: 8000
      protocol: http
    epp:
      pluginsConfigFile: "optimized-baseline-plugins.yaml"
      pluginsCustomConfig:
        optimized-baseline-plugins.yaml: |
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          plugins:
          - type: queue-scorer
          - type: kv-cache-utilization-scorer
          - type: prefix-cache-scorer
          - type: no-hit-lru-scorer
          schedulingProfiles:
          - name: default
            plugins:
            - pluginRef: queue-scorer
              weight: 2
            - pluginRef: kv-cache-utilization-scorer
              weight: 2
            - pluginRef: prefix-cache-scorer
              weight: 3          # prefix reuse gets the highest weight
            - pluginRef: no-hit-lru-scorer
              weight: 2

The one gotcha: the chart’s default epp and proxy resource requests are cpu: 4 / memory: 8Gi each - fine for a GPU box, unschedulable on a laptop. The demo overrides both down to 200m/512Mi. That override is the difference between the router pods running and sitting Pending forever.

Step 5: Send traffic and watch the routing#

task test:route port-forwards the router Service and fires requests through the Envoy proxy. Note the Service is named after the Endpoint Picker (llm-d-router-epp), not after the release - llm-d-router is the HelmRelease and the InferencePool.

The response carries an x-inference-pod header naming the pod the EPP chose, which is what makes the routing observable:

kubectl port-forward -n llm-d svc/llm-d-router-epp 8080:80 &

PROMPT="You are an SRE assistant with deep Kubernetes knowledge. Follow this runbook precisely: first restart the pod, then check the readiness probes, then verify the rollout status."

curl -sS -D - -o /dev/null http://localhost:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"Qwen/Qwen3-0.6B\",\"prompt\":\"$PROMPT\",\"max_tokens\":8}" | grep -i x-inference-pod

Fire that same prompt eight times, then eight distinct prompts, and tally which pod served each. This is the actual output from the Kind cluster:

8x identical prefix   ->  8  x-inference-pod: qwen-sim-6fbd95f76d-5njsd

8x distinct prompts   ->  2  x-inference-pod: qwen-sim-6fbd95f76d-xnmpc
                          2  x-inference-pod: qwen-sim-6fbd95f76d-jqxkw
                          2  x-inference-pod: qwen-sim-6fbd95f76d-65m85
                          2  x-inference-pod: qwen-sim-6fbd95f76d-5njsd

That is the whole thesis of the post in eight lines. Shared-prefix traffic collapses onto a single pod because the prefix-cache scorer knows that pod already holds the prompt’s prefix. Cold, unrelated traffic spreads perfectly evenly across all four because the no-hit LRU scorer takes over when no pod has a cache hit. A round-robin Service would have scattered the first set and, over eight requests, would not have guaranteed the even split on the second.

Then inspect the EPP’s metrics to see the scoring in action (task test:metrics). The metrics port is authenticated - it returns 401 without a bearer token, so mint one first:

TOKEN=$(kubectl create token default -n llm-d --duration=10m)
kubectl port-forward -n llm-d svc/llm-d-router-epp 9090:9090 &
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:9090/metrics \
  | grep -Ei 'inference_|prefix|score|queue|kv'

This needs a little RBAC that the router chart does not ship: system:auth-delegator on the EPP’s ServiceAccount so it can run a TokenReview, plus a /metrics reader role bound to the ServiceAccount whose token you use. Both live in kubernetes/metrics/rbac.yaml in the demo and are applied by task metrics:rbac. Without the first one the EPP answers 500 Authentication failed even with a valid token.

The scorer chain shows up by name in the output:

inference_extension_info{build_ref="v0.9.0",...} 1
inference_extension_plugin_duration_seconds_count{extension_point="DataProducer",plugin_name="approx-prefix-cache-producer",...} 20
inference_extension_plugin_duration_seconds_count{extension_point="PreRequest",plugin_name="no-hit-lru-scorer",...} 20
inference_extension_plugin_duration_seconds_count{extension_point="Picker",plugin_name="max-score-picker",...} 20

Note: every command and every output above was captured from a real run on Kind (sim v0.10.0, GAIE v1.5.0, router chart v0.9.0), on a laptop with no GPU. One gotcha worth pinning: the sim needs --force-dummy-tokenizer when you pass a real Hugging Face model name like Qwen/Qwen3-0.6B. Without it, the sim takes the HF tokenization path, tries to reach a render service on localhost:8082, and every model-server pod goes CrashLoopBackOff.

llm-d: Pros and Cons#

Pros#

Cons#

When to use llm-d#

Use it when:#

• You serve one or a few large models under real traffic - the GPU bill is dominated by a handful of models and you need to squeeze utilization
• Your prompts share long prefixes - RAG, long system prompts, few-shot templates - where prefix-cache-aware routing is a large latency win
• Prefill and decode profiles differ - long prompts with short outputs, or vice versa, where disaggregation lets you right-size each phase
• You run multi-node model replicas - LeaderWorkerSet orchestration is first-class
• You are standardizing on the Gateway API - llm-d rides the same GAIE that KServe and others are adopting

Consider alternatives when:#

• You serve a small model at low QPS - run vLLM directly behind a Service. llm-d’s machinery is dead weight here.
• You need a multi-framework serving platform - sklearn, XGBoost, ONNX, and LLMs through one abstraction - use KServe. Not instead of llm-d: KServe’s LLMInferenceService runs llm-d underneath, so you get both from one control plane.
• You want a turnkey managed operator - KAITO auto-provisions GPU node pools and deploys open-weight models with less assembly.
• Your bottleneck is GPU scheduling, not routing - start with Kueue or Volcano for fair-share and gang scheduling.

Troubleshooting#

Production considerations#

The demo swaps GPUs for a simulator. A real deployment differs in the ways that matter most.

Real GPUs and the engine#

In production the model-server pods run vllm/vllm-openai with nvidia.com/gpu limits, --enable-prefix-caching, --tensor-parallel-size for multi-GPU, and a --kv-transfer-config connector for disaggregation. A 32B model on a single H100 serves tens of tokens/sec per stream; disaggregation and cache-aware routing are what keep p99 flat as you add replicas. The simulator reproduces the routing behavior but not these throughput numbers.

Prefill/decode sizing#

Disaggregation only pays off if you actually split the pods and size them to your traffic. A RAG workload with 4k-token prompts and 200-token answers is prefill-heavy - provision more prefill capacity. A chat workload with short prompts and long answers is decode-heavy - do the opposite. Measure your prompt-to-output ratio before fixing the split.

KV-cache tiers#

LMCache’s offload hierarchy needs real backing storage in production: fast local NVMe for the CPU→disk tier, or a shared cache (Redis, an object store) if you want cache reuse across pods. Sizing these tiers is the difference between resuming a conversation instantly and recomputing it.

Autoscaling#

Scale the model-server pools on inference metrics - KV-cache utilization and queue depth from the EPP - not CPU. Pair with a node autoscaler (Karpenter or cluster-autoscaler) scaling the GPU node pool itself, and Kueue for fair-share across teams.

Cost#

This is where the story comes full circle. Distributed inference means a lot of expensive GPUs running a lot of the time, and cache-aware routing plus disaggregation exist largely to cut that bill. To see the bill - per model, per team, per GPU-hour - you need cost attribution. That is OpenCost territory, and it is the natural next post in this series: putting a dollar figure on the GPU inference platform we have been building across KServe and now llm-d.

Hands-On Demo Repository#

The full demo lives at github.com/nicknikolakakis/srekubecraft-demo/tree/main/llm-d. It includes:

• A Taskfile.yml with task up / task down to spin the whole thing up on Kind
• Flux GitRepository + Kustomization manifests for GitOps delivery
• The GAIE + Gateway API base, the llm-d-inference-sim model-server Deployment, the InferencePool, and the EPP HelmRelease
• The Mermaid diagrams from this post
• A README walkthrough with the captured routing metrics from a real Kind run

Quick start:

git clone https://github.com/nicknikolakakis/srekubecraft-demo
cd srekubecraft-demo/llm-d
task up      # creates the kind cluster, bootstraps flux, reconciles everything

Conclusion#

llm-d is the answer to a question KServe deliberately left open: once a large model has to serve real traffic across a fleet of GPUs, how do you route, disaggregate, and cache inference so tail latency stays flat and GPUs stay busy? A plain Kubernetes Service cannot, because the signals that matter - prefix-cache locality, KV-cache pressure, queue depth, the prefill/decode split - are invisible at the HTTP layer. llm-d surfaces them through the Gateway API Inference Extension and turns them into routing decisions.

It is a CNCF Sandbox project, so expect API churn and bring your own GPU plumbing. But the architecture is right, the backing is serious, and the fact that you can run the entire orchestration layer on a laptop with the vLLM simulator - no GPU, fully reproducible via Flux - tells you the maintainers care about the operational experience, not just the benchmarks.

If you are running one small model, use vLLM directly. If a handful of large models dominate your GPU bill and your p99 is creeping up under load, llm-d is the 2026 tool built for exactly that problem. And if you are already on KServe, this is not a second platform to adopt: LLMInferenceService runs llm-d underneath, so the routing behavior in this post arrives through a CRD you already know.

Next in this series: putting a cost on all of it with OpenCost.

If you found this useful, you might also enjoy my related posts:

• KServe - Production ML Serving on Kubernetes, from sklearn to LLMs
• Building a Kubernetes Agent with Google’s ADK for Go
• Dapr on Kubernetes - The Distributed Application Runtime