Inference

~/posts/modelplane-beyond-placement-routing-dynamo-gotchas 22 min · 4548 words

$ Modelplane Beyond Placement - Routing Policy, the Dynamo Stack, and Four Gotchas

// Part two of the Modelplane fleet inference series, covering the layer above the scheduler. ModelService is a routing policy rather than a load balancer, spec.stack Dynamo swaps the whole gang-scheduling machinery, and four documented gotchas will cost you an afternoon each.

~/posts/modelplane-fleet-inference 29 min · 5971 words

$ Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU

// A hands-on tour of Modelplane, the Crossplane-based control plane for AI inference across a fleet of GPU clusters. Two regions on Kind with fake DRA GPUs, the platform/developer contract written in DRA's own vocabulary, fleet-level drain, and the release trap that costs you two APIs.

~/posts/llm-d-distributed-inference 21 min · 4271 words

$ llm-d - Kubernetes-Native Distributed LLM Inference at Scale

// A hands-on tour of llm-d, the CNCF Sandbox framework for distributed LLM inference on Kubernetes - inference-aware routing, prefill/decode disaggregation, and KV-cache offload. Includes a GPU-free demo on Kind using the vLLM simulator, wired with Flux GitOps.

posts tagged #Inference