Modelplane Beyond Placement - Routing Policy, the Dynamo Stack, and Four Gotchas

The first Modelplane post had one job: prove the fleet scheduler on a laptop. Three Kind clusters, fake DRA GPUs, a mock engine, and a manifest that named no cluster and no region landing one replica in each. That is what the demo was built to show, and it showed it.

Everything sitting above the scheduler I parked, on purpose. A post that tries to cover placement, traffic policy, sizing, an alternate serving stack and the storage path in one sitting is a post nobody finishes. So this is the second half: the API surface you reach for once placement works and you have to run a release process on top of it.

I did go back through the docs and the XRDs line by line before writing it, and the first post’s evidence holds. Every scheduling rule, every capacity formula and both known limits still read the same. Nothing here is a retraction.

Provenance: everything below is read from the official docs (docs/content/ on main) and the XRD definitions (apis/*/definition.yaml). The first post’s evidence was hands-on. This one is docs and schema, and the last section is the exact set of things I am adding to the demo to close that gap.

Who Should Read This?#

This post is for:

• Anyone who read the first post and wants the layer above the fleet scheduler, which that post deliberately did not open
• Platform Engineers sizing a Modelplane deployment, who need the three independent numbers rather than the one the first post explained
• SREs planning a canary or a cluster drain on a GPU fleet, since the mechanism for both is the same object
• Anyone evaluating Modelplane on a multi-node topology, where the Dynamo stack and the NIXL requirement decide whether your engine starts at all

TL;DR#

• ModelService is a routing policy, not a load balancer. Endpoints carry modelplane.ai/deployment and modelplane.ai/cluster labels, entries combine, and weights are relative. That gives you canaries, per-cluster drain without a redeploy, and SaaS overflow behind the same URL.
• There are three sizing numbers, not two: spec.replicas, engines[].copies and worker.nodes. Only the first is a scaling axis. There is no in-cluster pod autoscaling.
• spec.stack: Dynamo swaps LeaderWorkerSet for Grove plus KAI Scheduler and adds ModelExpress, which pulls model weights from a peer replica over RDMA instead of from storage.
• Four gotchas: usable versus nominal GPU memory, a revision mismatch that silently re-downloads the model, PrefillDecode needing NIXL in your image, and a fabric field that is not Nebius-specific.
• Traefik is the only gateway backend for a documented reason: per-backend path rewriting in Gateway API.

What the first post left for this one#

Four threads, each parked for the same reason: the demo could not exercise them, and describing an API I had not driven would have diluted a post whose whole point was hands-on evidence.

Bottom line: the first post proved the layer that decides where a model runs. This one is about the layer that decides what happens to a request once it does, which is where the operational leverage turns out to be.

1. ModelService is a routing policy, not a load balancer#

This is the miss that matters operationally.

Modelplane composes one ModelEndpoint per replica, only once that replica is Ready, and withdraws it when the replica goes unhealthy. On each endpoint it stamps two labels that carry routing intent:

spec.endpoints is a list, and the entries combine: the service routes to every endpoint that any entry matches. Three patterns fall out of that.

Route to a whole deployment#

The case the first post showed, and the one you write 90% of the time:

apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b

Route to part of one#

Pair the deployment label with the cluster label and you have taken a cluster out of service without touching the deployment, without a taint, and without a redeploy:

spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b
        modelplane.ai/cluster: prod-us-east

Why this matters: the first post covered drain by taint, which is the heavy instrument. It moves replicas. This is the light one: the replicas stay exactly where they are and stop receiving traffic. Two different operations that the first post collapsed into one.

Split by weight#

Weights are relative, and they apply to the entry as a whole. The share spreads as evenly as possible across every endpoint that entry matches, so scaling a deployment up or down does not change its share of traffic. An entry with no weight defaults to 1:

spec:
  endpoints:
  - weight: 95
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b
  - weight: 5
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b-v2

That is a canary across a fleet, expressed in one namespaced object that a developer owns.

An entry can also select a hand-made ModelEndpoint pointing at an OpenAI-compatible SaaS provider, so overflow or break-glass traffic sits behind the same URL as your own replicas. The first post mentioned that a ModelEndpoint can be hand-made; it did not say that weights are how you make it useful.

Two things worth being precise about#

These are easy to conflate, and conflating them is how you oversell the system to your own team.

Placement is provider-aware; routing is policy you write. The fleet scheduler has no cost model, no provider preference and no cache weighting of its own. But because every endpoint carries modelplane.ai/cluster, you can express “90% Crusoe, 10% Nebius” as a ModelService, and Modelplane withdraws endpoints whose replicas go unhealthy. That is a static split with health-based withdrawal. It is not capacity-aware failover. Do not sell it as the latter.

Disaggregated serving is the one exception to API-shape transparency. Prefill and decode routing reads OpenAI-format request bodies to pick the pair, so a request in another API shape still reaches the engine but skips that cache-aware routing. Unified serving forwards every shape identically.

2. The third sizing number: engines[].copies#

The first post used nodes = pods x copies without explaining where copies comes from. There are three independent numbers, and only one of them scales:

copies buys in-cluster resilience: a node failure drops one copy instead of taking the replica out of service. In disaggregated serving it also sets the prefill-to-decode ratio, which is the number you actually tune for a given traffic mix.

Worth stating plainly, because it is the kind of thing you assume works and then discover under load: there is no in-cluster pod autoscaling. The deployment exposes the Kubernetes scale subresource, so kubectl scale and KEDA work against spec.replicas, but nothing scales pods inside a replica. If you want more throughput in one cluster, you add a replica and let the fleet scheduler decide where it goes, or you size copies correctly on day one.

3. The Dynamo stack#

Entirely absent from the first post, and it changes two of its conclusions.

An InferenceCluster can set spec.stack: Dynamo. That swaps the gang-scheduling machinery from LeaderWorkerSet to Grove plus KAI Scheduler, so a ModelReplica composes a Grove PodCliqueSet instead of an LWS, and the serving stack additionally runs a ModelExpress server.

flowchart TB
    subgraph Default["spec.stack: default"]
        LWS["LeaderWorkerSet"] --> LP["gang pods"]
        LP -->|"--load-format default"| PVC1["ModelCache PVC"]
    end

    subgraph Dyn["spec.stack: Dynamo"]
        G["Grove PodCliqueSet"] --> KAI["KAI Scheduler"]
        KAI --> DP["gang pods"]
        DP -->|"--load-format modelexpress"| MX["ModelExpress server"]
        MX -->|"peer over RDMA"| DP
        MX -.->|"fallback"| PVC2["ModelCache PVC"]
    end

ModelExpress changes the weight-distribution story from the first post. With --load-format modelexpress, the first replica loads from its PVC seed and publishes itself as a source; later replicas pull from a peer over RDMA rather than reading storage again.

The catch, and it is the one that decides your storage bill: a replica that finds no compatible peer, or no fabric to reach one over, falls back to the PVC. So the cache still has to be sized and staged for every replica. ModelExpress is a fast path, not a way to skip provisioning.

One operational detail that will confuse you in a debug session: under Dynamo, MODELPLANE_RANK is not how a pod learns its rank. The command derives it from Grove’s own GROVE_PCLQ_POD_INDEX. MODELPLANE_LEADER_ADDRESS resolves on both stacks, so that half of the first post’s “injects almost nothing” claim survives intact.

4. Four gotchas worth having#

The first post had five gotchas I hit by running the thing. These four are documented, or documented in a schema description, and they are the ones I would hit next.

Publish usable capacity, not nominal#

An 80GB H100 reports about 81559Mi of usable memory. Declare 80Gi on your InferenceClass and a nodeSelector asking for >= 80Gi will match the pool and then fail to bind the GPU. The fleet scheduler says yes, DRA admission on the workload cluster says no, and you get a placement that never becomes Ready.

This is the sharpest edge in authoring an InferenceClass, and unlike most of the first post’s gotchas it is actually documented. Publish what the driver reports, not what the marketing page says.

A revision mismatch downloads the model twice#

A bare repo ID resolves at the default branch. A ModelCache pinned to a commit or a tag needs the engine to pass that same revision (--revision for vLLM). An engine asking for the default branch finds nothing staged under it and quietly downloads the whole model again, which is precisely the cost the cache existed to remove.

No error, no warning. Just a cold start that takes as long as it did before you built the cache, and an egress bill.

PrefillDecode needs NIXL in your image#

vLLM’s NixlConnector and SGLang’s prefill and decode transfer both import the nixl package. On an image that lacks it, disaggregated engines crash at startup with NIXL is not available.

Recent vanilla vllm/vllm-openai tags include it. Modelplane does not bundle it, and that is consistent with the engine-neutrality the first post praised: the engine image is yours, so its dependencies are yours too.

There is a fabric field, and it is not just Nebius#

nodePools[].fabric.type selects the node-to-node fabric for multi-node engines, “so a gang’s tensor-parallel traffic isn’t capped by TCP”, defaulting to None. The XRD carries a validation rule requiring fabric.infiniband when the type is InfiniBand on a Nebius source, and there is an EFA DRA driver path on the AWS side.

This is the real counterpart to the claim: Synthetic InfiniBand example I used in the first post. Synthetic describes a fabric for placement; fabric.type configures the one the gang actually uses. I showed the first and never mentioned the second.

5. Why Traefik is the only backend#

The first post listed “Traefik-only gateway” as a con and left it there. The architecture docs give the reason, and the reason is a better answer than the complaint:

Each hop rewrites the path: the control plane rewrites the public prefix to the replica’s path, and the workload gateway strips that down to what the engine serves. This per-backend path rewriting is the main thing the control-plane gateway has to support, and it narrows which Gateway API implementations can fill the role.

So backend is an enum on purpose. The constraint is per-backend path rewriting in Gateway API, not a preference for Traefik. Both hops are Gateway API, and which implementation sits at each layer is explicitly internal and not part of the API.

That downgrades the con from “vendor lock-in in the data path” to “a narrow Gateway API feature has few implementations”, which is a genuinely different thing to be annoyed about.

6. What moved since v0.3.1#

The v1.35 line needs care, because it does not retire the standoff from the first post’s gotcha #3, and I initially assumed it did.

Two different clusters are involved. faq.md says each inference cluster needs “Dynamic Resource Allocation (DRA, Kubernetes v1.35+) to bind GPUs to pods”. But installation.md, which builds the control cluster, still pins kindest/node:v1.34.0 under the comment “kind v0.31+ ships containerd 2.2.0 which breaks Modelplane” (#315). The control cluster has no GPUs and needs no DRA, so those two coexist without contradiction.

The corner is still sharp for anyone running inference clusters on kind, which is exactly what my demo does. DRA wants v1.35+, and a kind v1.35 node image ships the containerd the install note warns about. I have not tested that combination, and it is the first thing I would ask upstream.

7. The one place the docs and the cluster disagree#

drain-cluster.md says of a drain that cannot finish:

Its ReplicasScheduled condition reports the shortfall, so a drain that can’t finish is visible rather than silent.

The first post said the status stays True with reason ReplicasCreated and the boolean never flips. Running the canary produced both cases side by side, and the truth is more specific than that:

$ kubectl -n ml-team get modeldeployment \
    -o jsonpath='...ReplicasScheduled...'
mock-demo     status=True   reason=ReplicasCreated       msg=Scheduled 1 of 2 replicas
mock-demo-v2  status=False  reason=InsufficientCapacity  msg=0 of 1 replicas scheduled (checked 2 clusters)

Same cluster, same taint, same moment. So:

The boolean does flip, but only at zero. A deployment running at half its requested replicas reports ReplicasScheduled=True, which is the case that actually happens during a drain and the case an alert would miss. My first post overstated this as “never flips”; the accurate version is that it is blind to partial shortfall specifically.

Alert on status.replicas.ready against spec.replicas, not on the condition. The upstream issue is still worth opening, with the sharper wording.

What still stands from the first post#

So this does not read as a retraction. Verified unchanged against the docs and the XRDs on main:

• The DRA contract thesis. It is now literally a docs section titled “The device contract”.
• claim: Synthetic, with the same InfiniBand example, in the InferenceClass XRD.
• Crossplane v2 XRs, no Go controller, Python composition functions.
• Every scheduling rule. Pure function of observed state, existing replicas as inputs and not decisions, two-level matching, nodes = pods x copies, zero-cost claimless members, the capacity ledger, an engine never split across pools, retain-then-fill with spread before pack, delete-plus-create on move.
• Both known limits, same issue numbers: #172 and #149.
• The BYO modelplane.ai/pool label trap, still documented only in passing.
• Vultr still has no usable RWX class on GPU nodes. HuggingFace is still the only cache source. Traefik is still the only gateway backend.
• Exclusive ownership: “dedicate each cluster to Modelplane rather than sharing it with other workloads”.
• Drain by taint with NoSchedule and NoExecute, tolerations to pin through one, and nothing rescheduling back after the taint is removed.

Proving it on the demo#

Everything above was docs and schema when I drafted it. Then I put the routing half on the cluster, because a post whose headline claim is untested is a worse post than one that waits a day.

The demo at srekubecraft-demo/modelplane-fleet proved placement and nothing above it. Here is what I added, and what it printed.

Before, the repo had one ModelDeployment and this ModelService:

# kubernetes/modelplane/50-model-service.yaml
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: mock-demo

One entry, no weight, no cluster label. That is the only routing shape the demo exercises.

1. A second deployment, for the weighted split#

The canary needs something to canary against, so the first addition is 41-model-deployment-v2.yaml: a copy of the mock at mock-demo-v2. No engine change is needed. The mock already answers served by <pod name>, and a pod name is prefixed with its deployment, so mock-demo-v2- in the response body is all the tally needs to tell the two apart.

Then 51-model-service-canary.yaml:

spec:
  endpoints:
  - weight: 90
    selector:
      matchLabels:
        modelplane.ai/deployment: mock-demo
  - weight: 10
    selector:
      matchLabels:
        modelplane.ai/deployment: mock-demo-v2

The capacity constraint bites here. Each region is a single control-plane node and capacity is charged per node, so the fleet has exactly two nodes. Two deployments at replicas: 2 do not fit.

Run both at replicas: 1 and it fits exactly, one node per region, which makes the canary a cross-region split as a bonus. That is what task canary does: it scales mock-demo down to 1 before applying v2.

The tempting alternative, adding a worker node, is three changes rather than one, and skipping any of them fails quietly:

Pay the RAM only if you want a replica count above one per region. For watching a weight, you do not.

First, the endpoint labels the whole section rests on, read off the control plane once both are Ready:

$ kubectl -n ml-team get modelendpoint \
    -o custom-columns=NAME:.metadata.name,DEPLOY:'.metadata.labels.modelplane\.ai/deployment',CLUSTER:'.metadata.labels.modelplane\.ai/cluster'
NAME                 DEPLOY         CLUSTER
mock-demo-be445      mock-demo      gpu-eu-west
mock-demo-v2-5ef67   mock-demo-v2   gpu-us-east

Both labels, on both endpoints, exactly as documented. And the fleet scheduler put v2 in the other region on its own, so the canary is a cross-region split for free.

2. A task that counts the split#

task curl sends one request. A weight needs a tally, so task curl:split loops inside a single in-cluster curl pod (the address is on the Kind subnet, and starting 200 pods would take longer than the test) and counts by deployment:

$ task curl:split N=200
==> 200 requests through http://172.18.255.200/ml-team/mock
mock-demo       180  ( 90.0%)
mock-demo-v2     20  ( 10.0%)
total           200

Exactly 90.0 and 10.0 on 200 requests. The headline claim of this post, no longer docs-only.

3. The experiment the docs do not answer#

With the tally in place, the open question becomes a five minute test. The obvious way to run it does not work:

$ kubectl -n ml-team scale modeldeployment mock-demo-v2 --replicas=0
The ModelDeployment "mock-demo-v2" is invalid: spec.replicas:
  Invalid value: 0: spec.replicas in body should be greater than or equal to 1

A ModelDeployment cannot be scaled to zero. The XRD floors spec.replicas at 1, so “park this deployment but keep it defined” is not a state the API has. Worth knowing on its own, and it also means the KEDA integration the docs mention can scale within a range but never to zero.

So the endpoint has to be withdrawn the other way, by making the replica unschedulable. Taint the cluster hosting v2 and it has nowhere to go, since the other region’s single node is already occupied:

task drain CLUSTER=gpu-us-east   # v2's replica is evicted, cannot be re-placed
task curl:split N=200            # the entry still exists, its endpoints do not

The endpoint is withdrawn within 20 seconds, leaving the weighted entry pointing at nothing. The answer:

$ kubectl -n ml-team get modelendpoint
NAME              DEPLOY
mock-demo-be445   mock-demo

$ task curl:split N=200
mock-demo       200  (100.0%)
mock-demo-v2      0  (  0.0%)
total           200

The weight redistributes. 200 of 200, and not one failed request. So “failover” is a fair word for the 90/10 pattern after all: the split is static, but an entry whose endpoints are all gone stops consuming its share rather than blackholing it. That is the behaviour you want and the docs never state it.

4. Traffic drain, next to the taint drain#

52-model-service-drain.yaml pins the service to one cluster:

spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: mock-demo
        modelplane.ai/cluster: gpu-us-east

Run it against the existing task drain and the contrast is the whole point: the taint moves replicas, the selector moves traffic and leaves the replicas running.

One thing to get right in the tooling, which I got wrong first: a tally by deployment cannot see this. Both replicas belong to mock-demo, so a deployment count reads 100% either way and proves nothing. task curl:split therefore tallies by replica, which carries both the deployment and, via ModelReplica, the cluster:

$ task curl:split N=60
==> 60 requests through http://172.18.255.200/ml-team/mock
mock-demo-49b71           60  (100.0%)
total                     60
==> replica -> cluster:
REPLICA           CLUSTER
mock-demo-49b71   gpu-us-east
mock-demo-be445   gpu-eu-west

60 of 60 to the us-east replica. The eu-west replica is still there, still Ready, still costing you a GPU, and receiving nothing. task traffic:undrain restores the unpinned service and the control comes back clean:

mock-demo-be445           30  ( 50.0%)
mock-demo-49b71           30  ( 50.0%)
total                     60

That is the distinction the first post could not draw, because it only had the taint.

5. The v1.35 bump#

The demo’s two workload clusters are inference clusters, and faq.md now wants v1.35+ on those for DRA. So the node image digest changes in kind-eu.yaml and kind-us.yaml, and the check is the one the first post already used:

kubectl --context kind-modelplane-eu-west get resourceslices
kubectl --context kind-modelplane-eu-west get resourceclaims   # allocated,reserved

This is the expensive one, not the cheap one. A kind v1.35 image ships the containerd that installation.md warns about, and UPSTREAM in the Taskfile is pinned to the Modelplane tag, so it also supplies the DRA driver manifest. Bumping the node image probably drags MODELPLANE_VERSION with it. hypothesis: untested, and the reason this row is still open.

6. copies, which is nearly free#

Set engines[].copies: 2 on the mock and the capacity ledger should demand two nodes per replica rather than one. On a single node per region that means one replica places and the second reports a shortfall, which is a cheap way to watch the formula from the first post do its arithmetic. It needs no new manifest, just a field and a task placement.

Where that leaves the post#

The last row is the ceiling, and it is worth naming precisely. Grove and KAI Scheduler would install, but ModelExpress pulling weights from a peer over RDMA needs a fabric that does not exist on a Docker network, and the mock engine has no weights to pull. NIXL needs a real vLLM image. The 81559Mi gotcha needs a real H100, not a fake driver reporting whatever I told it to.

Those four stay docs-only until someone runs them on real hardware. The routing layer, which is the part that actually changes how you operate a fleet, no longer does.

Two things the run cost me before it produced anything#

Both are Kind-path operational notes rather than Modelplane bugs, and both cost me an hour, so they are worth writing down.

The clusters had been sitting untouched for two days. Every engine pod was in CreateContainerError:

CDI device injection failed: unresolvable CDI devices
k8s.gpu.example.com/gpu=89bece99-c997-4590-9f24-5f81e15d9df6-gpu-0

The DRA driver had restarted at some point, regenerating its CDI device names, while the pods’ ResourceClaim objects still referenced UUIDs from the previous driver instance. The claim reads allocated,reserved and containerd cannot resolve the device. Deleting the pods clears it, because the generated claim goes with them.

Then the replacements sat in ContainerCreating with a single Scheduled event and the driver logging numClaims=0 on every poll, which is gotcha #3 from the first post exactly: the kubelet 1.34.0 DRA deadlock, triggered by the 30 minute gRPC idle timeout that a cluster left overnight always hits. docker exec <node> systemctl restart kubelet on both workload nodes, and both engines were Running inside two minutes.

A parked Kind fleet does not come back on its own. Budget the kubelet restart before you budget the experiment.

Conclusion#

The one thing to carry away from this half: ModelService is where your traffic policy lives. Endpoint labels for deployment and cluster, entries that combine, and relative weights give you a canary, a per-cluster traffic drain, and SaaS overflow, all in one namespaced object a developer owns. That is what turns Modelplane from a placement engine into something you could run a release process on, and it is why it earned a post of its own instead of a paragraph in the last one.

The rest is calibration. engines[].copies is the sizing number nobody explains, spec.replicas being the only scaling axis is the kind of constraint you want before you plan capacity rather than after, and it has a floor of 1, so parking a deployment at zero is not a state this API has. The Dynamo stack is a genuine fork in the architecture rather than a flag.

And the four gotchas share a shape with the first post’s five: every one of them fails quietly. Nominal capacity binds nothing, a revision mismatch re-downloads silently, a missing NIXL crashes at startup, and ReplicasScheduled reports True through a partial shortfall. That last one is the pattern in miniature. This project’s failure mode is not an error, it is a green status next to a message nobody parsed.

For a v1alpha1 project the summary from the first post stands unchanged: evaluate it now on the laptop path for the design ideas, do not put production traffic behind it yet. The difference is where I would spend the evaluation. Placement behaves exactly as documented and the demo already proves it, so point your afternoon at ModelService instead, with the six additions above. Five of them run on the same three Kind clusters you already have.

If you found this useful, you might also enjoy my related posts on AI infrastructure and platform design:

• Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU
• llm-d - Distributed Inference on Kubernetes
• KServe - Model Serving on Kubernetes
• Understanding the Kubernetes Gateway API
• Introduction to Crossplane