posts tagged #Inference

~/posts/llm-d-distributed-inference 21 min · 4271 words
$ llm-d - Kubernetes-Native Distributed LLM Inference at Scale

// A hands-on tour of llm-d, the CNCF Sandbox framework for distributed LLM inference on Kubernetes - inference-aware routing, prefill/decode disaggregation, and KV-cache offload. Includes a GPU-free demo on Kind using the vLLM simulator, wired with Flux GitOps.

#sre #kubernetes #llm-d #vllm #inference #llm #genai #ml-serving #cncf #cloudnative #gitops #flux #gateway-api