Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU

Sun, 30 Aug 2026 00:00:00 +0000

Serving one model on one cluster is a solved problem. Pick an engine, hand it a GPU, point traffic at it. I have written about that path twice already, with KServe and with llm-d.

The version nobody has solved cleanly is the fleet. GPUs are scarce, so you take capacity wherever you can get it: some on EKS, some on GKE, a reserved block in a neocloud, a rack you already own. Now you have five clusters that each need the identical serving stack, and a permanent matchmaking problem deciding which model runs on which hardware in which region. Underneath that sit two different jobs that most platforms blur together: the platform team’s job of publishing what hardware exists, and the ML team’s job of asking for the hardware they need.

Modelplane on SREKubeCraft | Nick Nikolakakis

Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU