Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU

Serving one model on one cluster is a solved problem. Pick an engine, hand it a GPU, point traffic at it. I have written about that path twice already, with KServe and with llm-d.

The version nobody has solved cleanly is the fleet. GPUs are scarce, so you take capacity wherever you can get it: some on EKS, some on GKE, a reserved block in a neocloud, a rack you already own. Now you have five clusters that each need the identical serving stack, and a permanent matchmaking problem deciding which model runs on which hardware in which region. Underneath that sit two different jobs that most platforms blur together: the platform team’s job of publishing what hardware exists, and the ML team’s job of asking for the hardware they need.

Modelplane is Upbound’s answer, and the reason I spent a weekend on it is not the product. It is the contract mechanism. Modelplane uses Kubernetes Dynamic Resource Allocation vocabulary as the API surface between a platform team and its consumers, across cluster boundaries. That idea generalises well past GPUs.

I ran the whole thing on a laptop with no GPU and no cloud account.

Who Should Read This?#

This post is for:

• Platform Engineers who have to hand ML teams a self-service surface for GPUs without becoming the ticket queue for every model
• SREs running more than one GPU cluster who are tired of installing the same serving stack by hand in every region
• Anyone evaluating DRA as a platform contract rather than just as the replacement for the NVIDIA device plugin
• Crossplane users curious what a serious v2 Configuration looks like in the wild

TL;DR#

• Modelplane is an Apache-2.0 control plane for AI inference across a fleet of clusters, created by Upbound, built entirely on Crossplane v2 compositions. There is no operator of its own.
• The API splits by ownership: cluster-scoped kinds are platform-owned, namespaced kinds are developer-owned. That split is the RBAC boundary, not just documentation.
• The platform publishes a device vocabulary in DRA’s own terms; developers write DRA-shaped CEL against it; real DRA admission on the workload cluster stays authoritative.
• It runs GPU-free on Kind using source: Existing plus the upstream dra-example-driver, so you can evaluate the whole model for the cost of some RAM.
• It is v1alpha1 and says so. I hit a release-packaging trap that silently costs you two of the fourteen APIs.

The problem shape#

Two facts drive everything about this design.

GPUs are scattered. Capacity is wherever you could get it: different clouds, different regions, on-prem, a neocloud with a capacity reservation. You do not get to consolidate.

The models worth serving at this scale do not fit on one machine. So a “replica” is not a pod. It is a gang of pods that must land together, coordinating over an interconnect fabric.

Put those together and the platform team inherits a permanent matchmaking job. Modelplane’s whole shape is built around keeping that job separate from the developer’s.

What is Modelplane?#

Modelplane is one control cluster sitting above a fleet of GPU clusters, with a single OpenAI-compatible front door. Apache 2.0, created by Upbound, with a stated intent to move to a neutral foundation. Status is v1alpha1 and the README says “early development” without hedging.

The API group is modelplane.ai/v1alpha1, and every kind is a Crossplane v2 XRD (apiextensions.crossplane.io/v2) with a matching composition under apis/<plural>/. There is no Go controller with _types.go anywhere in the repo. The API is the composition layer, and all the logic lives in fourteen Python composition functions packaged inside the Configuration.

The API, split by role#

The hierarchy is deliberately core-Kubernetes one scope up: ModelDeployment to ModelReplica to ModelService to ModelEndpoint parallels Deployment to Pod to Service to Endpoint, across a fleet instead of within one cluster.

Notice what is cluster-scoped. Exactly the platform-owned kinds. Everything a developer touches is namespaced. Scope equals ownership, which means the ownership model is enforced by RBAC you already know how to write.

Bottom line: if you have ever tried to hand developers a GPU self-service API, you have written a t-shirt-size enum (gpu: large) and then maintained a translation table from it to real hardware. Modelplane’s bet is that you should not invent the vocabulary at all.

The DRA pattern, the part worth stealing#

This is the idea I would keep even if the project disappeared tomorrow.

An InferenceClass is the platform team’s declaration of what a pool offers. Each entry mirrors what a DRA driver publishes in a ResourceSlice, one entry per kind of device rather than per physical device:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: synthetic-gpu
spec:
  description: "Fake GPU via dra-example-driver (no real hardware)"
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.example.com
    deviceClassName: gpu.example.com
    count: 1
    capacity:
      memory: { value: "80Gi" }

The developer side then writes CEL over those same attributes and capacity, in a ModelDeployment:

nodeSelector:
  devices:
  - name: gpu
    count: 1
    selectors:
    - cel: |
        device.capacity["gpu.example.com"].memory.compareTo(quantity("20Gi")) >= 0

The upstream docs are explicit that this is “the same expression an ML engineer would write in a DRA ResourceClaim”. So the pattern is:

The platform publishes a vocabulary in DRA’s own terms. Consumers write DRA-shaped CEL against it. The control plane matches the two, and real DRA admission on the workload cluster remains authoritative.

Three properties make this better than the enum:

1. The vocabulary is not invented. It is what a real driver already reports on a real node, so the abstraction cannot drift from the hardware and needs no translation table.
2. The contract is discoverable. The keys a platform team puts on a class are the contract. A nodeSelector matches a pool only if the class publishes what it asks for. Adding an attribute extends the contract with no API change.
3. One expression works at both altitudes. A fleet-level filter and a cluster-level ResourceClaim are written identically, so there is one thing to learn.

claim: Synthetic, the honest escape hatch#

Devices can be marked claim: Synthetic, meaning the scheduler considers them for placement but never claims them. It exists for hardware that matters for placement but has no DRA driver yet, an InfiniBand fabric being the obvious case.

This is the pragmatic move that makes the pattern shippable in 2026. DRA drivers do not exist for everything, but placement still depends on that hardware. Rather than bolting on a second parallel mechanism for undriven hardware, they keep one vocabulary and flag which entries are claimable. Worth copying: when adopting an emerging upstream contract, add a flag for “described but not enforced” rather than a second schema.

One sharp edge, which cost me a first attempt: an engine whose only device is Synthetic is rejected by the scheduler. You need at least one claimable device.

How it schedules#

The fleet scheduler is a pure function of observed state. It recomputes the whole placement from scratch on every reconcile, reading the deployment, every InferenceCluster with published capacity, and every existing ModelReplica.

The consequence that matters operationally: existing replicas are inputs, not decisions. A healthy replica is never moved to improve the global picture, even if a better cluster shows up later. Placement does not churn under a running deployment.

Matching happens in two stages:

1. Clusters are filtered by clusterSelector.matchLabels against ordinary Kubernetes labels on each InferenceCluster: region, tier, provider, compliance posture. Organisational metadata, so string equality is enough.
2. Pools are filtered by the DRA device matching above.

Then the workload cluster’s own scheduler binds actual GPUs via DRA. Splitting it this way is the key design call. The control-plane scheduler answers “could this cluster plausibly host this replica”, not “which exact GPU does each pod get”. Device-level contention is left to DRA admission on the workload cluster, which is authoritative, and the next reconcile sees the result.

Four rules worth committing to memory:

• Capacity is charged in nodes, not GPUs. nodes = pods × copies. A member that claims no DRA device costs zero nodes.
• An engine is never split across pools. Gang members coordinate over their pool’s fabric, the scheduler cannot reason about fabric, so pool identity is the finest grain it has. A split gang risks landing on two fabrics, the collective never forms, and it hangs with no clear error.
• Placement is pinned, not advisory. Every scheduled pod gets a nodeSelector on the modelplane.ai/pool node label. Without it the cluster scheduler could put the pod on any matching pool and the fleet’s accounting would drift from reality.
• Retain, then fill. Existing replicas keep their cluster. Shortfall is placed one at a time onto the eligible cluster hosting the fewest of this deployment’s replicas: spread before pack.

A replica never changes cluster. Moving is always delete-plus-create, mirroring how Kubernetes treats a pod whose node is gone.

Known limits, both fail-safe#

Both under-count capacity rather than overcommit it. That is the right direction to be wrong in.

What Modelplane is NOT#

• Not an inference engine. It runs vLLM or whatever you point it at, and injects almost nothing: just the address a multi-node leader is reachable at, so workers can join. Parallelism, quantisation and KV transfer are engine flags you write yourself.
• Not a replacement for KServe or llm-d. Those serve models on a cluster. This places models across clusters and installs that stack for you.
• Not a standalone operator. It is a Crossplane Configuration. No Crossplane, no Modelplane.
• Not production ready. v1alpha1, by its own admission.

Modelplane vs the alternatives#

Bottom line: Modelplane sits a layer above llm-d and KServe rather than competing with them. If you have one cluster, you do not need it. The moment you have three, the matchmaking problem it solves is the one actually eating your week.

The install footprint, honestly#

This is where adoption cost hides. Modelplane ships as a Crossplane Configuration (xpkg.upbound.io/modelplane/modelplane) and the docs state plainly that “Crossplane provides the reconciliation engine and package management”. The Configuration pulls sixteen provider packages as dependencies, most of which exist only to provision clusters you may never provision.

Crossplane is required on the control cluster only. Inference clusters know nothing about it; they receive ordinary Helm releases and Kubernetes objects through provider-helm and provider-kubernetes, authenticated by a kubeconfig in a secretRef. So a BYO fleet does not get Crossplane imposed on its production clusters.

The honest cost for a shop not already on Crossplane is therefore not the inference layer. It is taking on a full Crossplane v2 estate, with packages, provider upgrades, and a second reconciliation engine to operate, alongside whatever IaC and GitOps stack you already run.

For a laptop, there is a trim worth knowing about. Upstream’s lean-control-plane.yaml scales the cloud providers to zero with an ImageConfig, and the ordering is the whole trick: an ImageConfig binds at ProviderRevision creation, so it has to be applied before the Configuration. Get that right and the entire control plane is 15 pods.

How a request finds its replica#

The single front door is the part that sounds like magic, so here is the actual chain, read off the composed objects on my control plane rather than the docs.

The InferenceGateway is a singleton named default: one Traefik on the control cluster, with a LoadBalancer address that becomes the host of every ModelService URL, http://<address>/<namespace>/<service>. That is the string you hand to ML teams.

Below it, for every replica, Modelplane composes a ModelEndpoint whose URL is the workload cluster’s own gateway, not a pod:

$ kubectl -n ml-team get modelendpoint
NAME              URL
mock-demo-49b71   http://172.18.255.120/ml-team/mock-demo-49b71/v1
mock-demo-be445   http://172.18.255.100/ml-team/mock-demo-be445/v1

.100 and .120 are the MetalLB addresses of the Envoy gateways on my EU and US clusters. On the control plane, each endpoint becomes a selectorless Service plus a hand-built EndpointSlice whose only endpoint is that remote address:

$ kubectl -n ml-team get endpointslice -o custom-columns=NAME:.metadata.name,ENDPOINTS:.endpoints[*].addresses[*]
NAME                     ENDPOINTS
mock-demo-43164b07c3f8   172.18.255.100
mock-demo-46d87a8261b4   172.18.255.120

An HTTPRoute on Traefik then fans /ml-team/mock across those Services. So the full path is:

flowchart LR
    C["client"] -->|"POST /ml-team/mock/v1/chat/completions"| T["Traefik<br/>control cluster<br/>.255.200"]
    T -->|"HTTPRoute"| S1["selectorless Service<br/>EndpointSlice → .255.100"]
    T -->|"HTTPRoute"| S2["selectorless Service<br/>EndpointSlice → .255.120"]
    S1 --> E1["Envoy AI Gateway<br/>eu-west"]
    S2 --> E2["Envoy AI Gateway<br/>us-east"]
    E1 -->|"InferencePool + EPP"| P1["engine pod"]
    E2 -->|"InferencePool + EPP"| P2["engine pod"]

Two things follow from that design:

• There is no cluster mesh. No Cilium ClusterMesh, no Submariner, no Istio multi-primary. The only requirement is that the control cluster can reach each workload cluster’s gateway address over plain IP. In the cloud that is a routable load balancer; on my laptop it is three kind clusters sharing one Docker network, which is why every MetalLB pool had to sit inside that network’s real subnet. Get that wrong and the request just times out.
• The inference-aware routing lives on the workload cluster, not the front door. Traefik only picks a cluster. Once the request lands, the serving stack’s Envoy AI Gateway and the Gateway API Inference Extension (InferencePool plus an endpoint-picker pod per replica, which you can see running next to the engine) do the model-aware part. That is the same machinery I covered in the llm-d post, and it explains why Modelplane and llm-d are layers, not rivals.

ModelEndpoint can also be created by hand to point at an OpenAI-compatible SaaS provider, and a ModelService treats it identically. That gives you overflow to a provider when the fleet is busy, or a break-glass failover, behind the same URL.

Where the weights come from#

The obvious question for a fleet: when the scheduler drops a replica onto a cluster in another region, how do 100 GB of weights get there?

Two paths, and the default is the boring one. Without a ModelCache, the engine fetches the model itself at startup, straight from the source, with the credential as HF_TOKEN in the engine’s env. Every pod start pays the full download.

A ModelCache changes that to once per cluster:

apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen3-Coder-480B-A35B-Instruct
    authSecret:
      name: hf-token
    sizeGiB: 1100

Modelplane hydrates a ReadWriteMany PVC on each cluster the cache is staged to, propagates authSecret to those clusters, and mounts the result at /mnt/models in every serving pod that references the cache. The engine reads locally: --model=/mnt/models. For a multi-node gang this is the difference between one copy per cluster and one copy per pod, so it is the recommended path there and optional for single-node cold starts.

The details that will bite you in production:

My demo skips the cache entirely because the mock engine has no weights, which is honest about what the demo does and does not prove. The scheduling and routing evidence above is real; the storage path is docs-only here.

The GPU-free demo: two regions on Kind#

Upstream ships an e2e suite that runs the whole control plane with no GPU and no cloud account, and I want to give credit for that: it is what made this post possible for the price of some RAM. But it uses one workload cluster, so it never exercises the thing Modelplane exists for. The demo below adds a second region and puts the fleet scheduler on the stand.

Architecture of the demo#

flowchart TB
    subgraph CP["kind: modelplane-control"]
        XP["Crossplane 2.4.0"]
        CFG["Configuration modelplane v0.3.1<br/>14 composition functions"]
        TR["InferenceGateway<br/>Traefik + MetalLB .255.200"]
    end

    subgraph EU["kind: modelplane-eu-west (k8s 1.34)"]
        SEU["ServingStack<br/>Envoy AI Gateway .255.100"]
        DEU["dra-example-driver<br/>fake gpu.example.com, 80Gi"]
        REU["ModelReplica"]
    end

    subgraph US["kind: modelplane-us-east (k8s 1.34)"]
        SUS["ServingStack<br/>Envoy AI Gateway .255.120"]
        DUS["dra-example-driver<br/>fake gpu.example.com, 80Gi"]
        RUS["ModelReplica"]
    end

    DEV["ModelDeployment<br/>replicas: 2<br/>CEL: memory >= 20Gi"] --> CFG
    CFG -->|"place"| REU
    CFG -->|"place"| RUS
    TR -->|"HTTPRoute"| SEU
    TR -->|"HTTPRoute"| SUS
    DEU -.->|"ResourceSlice"| REU
    DUS -.->|"ResourceSlice"| RUS

Three kind clusters on one Docker network. Two upstream primitives make it cloud-free:

The engine is a mock: 30 lines of Python stdlib answering /v1/chat/completions and /v1/messages the way vLLM does, and replying with its own pod name so every request through the front door is traceable to a region.

Prerequisites#

• Docker with 24 GB of memory. Three kind clusters and two full serving stacks (cert-manager, Envoy Gateway, Envoy AI Gateway, GAIE, LeaderWorkerSet, NFD, kube-prometheus-stack, the NVIDIA DRA driver) do not fit in the 8 GB default. The Taskfile refuses to start under 20 GB.
• kind, kubectl, helm, jq, task. No crossplane CLI, no Nix.
• Workload clusters pinned to Kubernetes v1.34 for the resource.k8s.io (DRA) APIs, GA there.

Step 1: Three clusters, two of them with fake GPUs#

task clusters    # three kind clusters, node image pinned by digest
task workloads   # MetalLB + dra-example-driver + pool label on both regions

Each region gets a MetalLB pool inside the detected kind subnet, disjoint from the other region’s and from the gateway’s. Then the fake driver, and the one label a BYO cluster owes Modelplane:

kubectl label node modelplane-eu-west-control-plane modelplane.ai/pool=gpu-eu-west

Skip that label and every engine pod for the pool stays Pending forever with no error. It is the sharpest edge in the BYO path, and it is documented only in the schema description.

Proof the DRA path is real before Modelplane ever sees it:

$ kubectl --context kind-modelplane-eu-west get resourceslices
NAME                                                           DRIVER            NODE
00000-gpu.example.com-modelplane-eu-west-control-plane-2xc4f   gpu.example.com   modelplane-eu-west-control-plane

Step 2: The control plane, trimmed#

task controlplane

Which is: Helm-install Crossplane 2.4.0, apply upstream’s lean-control-plane.yaml first, then the RBAC prerequisites, then the Configuration. The trim is an ImageConfig that maps every cloud provider family to a zero-replica DeploymentRuntimeConfig:

apiVersion: pkg.crossplane.io/v1beta1
kind: ImageConfig
metadata:
  name: dormant-cloud-providers
spec:
  matchImages:
  - { type: Prefix, prefix: xpkg.upbound.io/upbound/provider-aws- }
  - { type: Prefix, prefix: xpkg.upbound.io/upbound/provider-azure- }
  - { type: Prefix, prefix: xpkg.upbound.io/upbound/provider-gcp- }
  - { type: Prefix, prefix: xpkg.upbound.io/upbound/provider-family- }
  - { type: Prefix, prefix: xpkg.upbound.io/upbound/provider-nebius }
  - { type: Prefix, prefix: xpkg.upbound.io/upbound/provider-vultr }
  runtime:
    configRef:
      name: scale-to-zero

Order matters because an ImageConfig binds at ProviderRevision creation. Applied after the Configuration, the ten cloud controllers are already running and stay that way. Applied before, the whole control plane is 15 pods: Crossplane, its RBAC manager, the 14 composition functions, and the two providers the BYO path actually uses.

One thing I do differently from the docs: I pin the Configuration myself instead of applying upstream’s manifest. Gotcha #1 explains why.

apiVersion: pkg.crossplane.io/v1
kind: Configuration
metadata:
  name: modelplane
spec:
  package: xpkg.upbound.io/modelplane/modelplane:v0.3.1

Step 3: Register the fleet#

task register

The platform team’s side of the contract, three objects. The class:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: synthetic-gpu
spec:
  description: "Fake GPU via dra-example-driver (no real hardware)"
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.example.com
    deviceClassName: gpu.example.com
    count: 1
    capacity:
      memory: { value: "80Gi" }

Two clusters that publish it, distinguished only by a label:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gpu-eu-west
  labels:
    modelplane.ai/region: eu-west
spec:
  cluster:
    source: Existing
    existing:
      secretRef:
        name: eu-west-kubeconfig
        key: kubeconfig
  nodePools:
  - name: gpu-eu-west
    className: synthetic-gpu
    nodeCount: 1
---
# gpu-us-east is identical apart from the name, label and secret

And the gateway, with loadBalancer: MetalLB so the composition installs MetalLB on the control plane itself. The kubeconfig secrets come from kind get kubeconfig --internal, because the control plane’s provider pods have to reach the workload API servers across the Docker network, not via 127.0.0.1.

This is the slow step. Each InferenceCluster gets the full serving stack pushed through provider-helm; on my laptop the two clusters went from 7 namespaces to 15 and reported Ready about 25 minutes later:

$ kubectl get inferencecluster
NAME          SOURCE     GATEWAY          SYNCED   READY
gpu-eu-west   Existing   172.18.255.100   True     True
gpu-us-east   Existing   172.18.255.120   True     True

Those gateway addresses are exactly the MetalLB pools I assigned, and they are what the front door will route to.

Step 4: The developer’s side, and where it lands#

task deploy

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: mock-demo
  namespace: ml-team
spec:
  replicas: 2
  template:
    spec:
      engines:
      - name: mock
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.example.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: python@sha256:6d43704baacd1bfbe7c295d7f13079d5d8104ed33568873133f8fc69980419df
                command: ["python", "-u", "-c"]
                args: ["..."]   # the mock server, in the repo

Read what is not in there: no cluster, no region, no cloud, no node pool. Just the shape of the hardware, as CEL over the same attributes the class published. The nodeSelector is deliberately a curated subset of a pod spec: the composition wires the port, readiness probe and the DRA ResourceClaim itself, and the XRD rejects ports or resources if you try to add them.

Thirty seconds later:

$ task placement
==> Replicas by cluster:
REPLICA           CLUSTER       READY
mock-demo-49b71   gpu-us-east   True
mock-demo-be445   gpu-eu-west   True

==> ReplicasScheduled:
True  ReplicasCreated  Scheduled 2 of 2 replicas

==> ModelService address:
http://172.18.255.200/ml-team/mock

One replica per region, from a manifest that named neither. That is the spread before pack rule from the scheduling docs, observed rather than quoted. On the workload side, the ResourceClaim binds a fake device through the real allocation path:

$ kubectl --context kind-modelplane-eu-west get resourceclaims
NAME                                                        STATE                AGE
mock-demo-be445-mock-47cef-6db546b4c9-cgm87-devices-hrk68   allocated,reserved   11m

Step 5: One front door, two regions#

$ for i in 1 2 3 4 5 6; do task curl; done
served by mock-demo-be445-mock-47cef-6db546b4c9-cgm87
served by mock-demo-49b71-mock-995c3-7fdcf68c88-pbbxl
served by mock-demo-be445-mock-47cef-6db546b4c9-cgm87
served by mock-demo-49b71-mock-995c3-7fdcf68c88-pbbxl
served by mock-demo-be445-mock-47cef-6db546b4c9-cgm87
served by mock-demo-49b71-mock-995c3-7fdcf68c88-pbbxl

Same URL, alternating regions. The client never learns there are two clusters. That curl runs from a pod on the control plane because the address is on the kind subnet, which a macOS host cannot route to.

Step 6: The drain, the point of a fleet#

Tainting an InferenceCluster is the fleet-level kubectl drain. NoSchedule stops new placements; NoExecute also moves what is already there. I scaled to one replica first so the move has somewhere to go, since each region has one node and capacity is charged per node.

$ kubectl -n ml-team scale modeldeployment mock-demo --replicas=1
$ task drain CLUSTER=gpu-eu-west
Tainted gpu-eu-west NoExecute.

Twenty seconds later the EU replica is gone and a new one is being created in US; a minute after that it is serving:

--- immediately after taint ---
REPLICA           CLUSTER       READY
mock-demo-49b71   gpu-us-east   False

--- after move ---
REPLICA           CLUSTER       READY
mock-demo-49b71   gpu-us-east   True

Delete-plus-create, exactly as the docs say, and the same way Kubernetes treats a pod whose node is gone. The replica name that came back was identical to the one dropped at scale-down, so the name appears to be derived from the placement, not a counter.

Now the honest failure. Scale back to two while EU is still tainted, and there is nowhere to put the second replica:

$ kubectl -n ml-team scale modeldeployment mock-demo --replicas=2
--- with one cluster drained ---
REPLICA           CLUSTER       READY
mock-demo-49b71   gpu-us-east   True
ReplicasScheduled=True  ReplicasCreated: Scheduled 1 of 2 replicas

The deployment runs below spec.replicas and the condition says so. One quibble worth knowing: the message tells the truth, but the condition’s status stays True with reason ReplicasCreated. If you alert on that boolean you will miss a drain that cannot finish; parse the message or compare replica counts instead.

Remove the taint and the fill phase does its job:

$ task undrain CLUSTER=gpu-eu-west
--- after undrain ---
REPLICA           CLUSTER       READY
mock-demo-49b71   gpu-us-east   True
mock-demo-be445   gpu-eu-west   True
ReplicasScheduled=True  ReplicasCreated: Scheduled 2 of 2 replicas

Nothing moved back. The US replica stayed put, because a taint governs new placement only, and the shortfall was filled onto the now-eligible cluster. Retain, then fill.

The Gotchas#

Five, and I hit every one of them. Two are Modelplane’s, one is Kubernetes’, two are the BYO path’s.

1. The install docs hand you the previous release#

docs/manifests/getting-started/configuration.yaml lags one release at every tag. The copy at tag v0.3.1 still pins the package v0.3.0; only main pins v0.3.1. Follow the install docs from a tag and you get the release before the one you checked out, without being told.

It matters because v0.3.0’s package ships a CEL rule the API server rejects. The label-key regex was written as \\. in double-quoted YAML, which reaches CEL as \., an invalid escape in a CEL string literal. Two of the fourteen XRDs never establish:

EstablishComposite: cannot apply rendered composite resource CustomResourceDefinition:
  "modelreplicas.modelplane.ai" is invalid: x-kubernetes-validations[1].rule:
  compilation failed: ERROR: <input>:1:51: Syntax error: token recognition error

The failure is silent. The XRD carries no status conditions at all, nothing lands in the Crossplane logs, and the only downstream symptom is an InferenceCluster stuck at SYNCED=False with no matches for kind "ModelReplica". The evidence lives in events:

kubectl get events -A --field-selector type=Warning | grep EstablishComposite

Fixed in #400, backported in #403. The v0.3.1 release notes call it “a critical fix for XRD creation” and add, tellingly, that v0.3.1 “is now usable on fresh installations”. Pin the Configuration yourself.

2. Do not upgrade v0.3.0 to v0.3.1 in place#

“Fresh installations” is doing work in that sentence. v0.3.1 moved provider-helm and provider-kubernetes from the modelplane fork to the upbound org (#390). Upgrade in place and both are installed:

upbound-provider-helm  Healthy=False  UnhealthyPackageRevision:
  cannot establish control of object: releases.helm.m.crossplane.io is already
  controlled by ProviderRevision modelplane-provider-helm-840f6120b7c2

The old fork keeps the CRDs so the new provider never starts, while the RBAC in prerequisites.yaml is bound to the new provider’s service account. The old one runs unprivileged and the gateway fails with namespaces is forbidden. Start clean.

3. Engine pods stuck in ContainerCreating with no events: kubelet 1.34.0#

This one cost me an afternoon and is not Modelplane’s fault. Both engine pods sat in ContainerCreating for hours. The ResourceClaim said allocated,reserved. The pod had exactly one event, Scheduled. The kubelet logged nothing. The DRA driver had never received a NodePrepareResources call carrying a claim.

A kubelet goroutine dump (/debug/pprof/goroutine?debug=1 through the node proxy) showed the pod worker blocked here:

dra.(*Manager).PrepareResources
  → plugin.(*DRAPlugin).NodePrepareResources
    → grpc.invoke → idle.(*Manager).ExitIdleMode        [blocked]

while a second goroutine held the same idle manager from inside the connection-close path:

http2Client.Close
  → plugin.(*monitoredPlugin).HandleConn   dra_plugin_manager.go:118
    → grpc.(*ClientConn).Connect → idle.(*Manager).ExitIdleMode   [blocked]

The kubelet’s DRA plugin monitor reconnects from inside the callback that fires when the connection ends, and when the connection ends because it went idle, Connect() deadlocks against the idle manager. The default gRPC idle timeout is 30 minutes, so a cluster used continuously never sees it and a cluster left overnight always does. Upstream fix: kubernetes/kubernetes#133926, cherry-picked to 1.34 in #133934, shipped in v1.34.2.

Workaround on an affected node: docker exec <node> systemctl restart kubelet. Both pods went Running within 75 seconds.

The catch is that Modelplane’s docs pin kindest/node:v1.34.0 because newer kind images ship containerd 2.2+, which upstream says “breaks Modelplane” (#315, no detail). So the documented image has the kubelet bug and the fixed image has an undocumented containerd concern. v1.34.8 carries containerd 2.3.1; I have not yet tested it against Modelplane.

4. A Synthetic-only engine is rejected#

claim: Synthetic is for hardware that matters for placement but has no DRA driver, and it works exactly as advertised for a second device. But an engine whose only device is Synthetic is refused by the fleet scheduler. You need at least one claimable device, which is the whole reason the fake driver is in the demo.

5. The BYO label, and the subnet#

On a BYO cluster Modelplane does not label your nodes. Without modelplane.ai/pool=<nodePools[].name> on the pool’s nodes, engine pods stay Pending with no error. And every MetalLB pool across all three clusters must sit inside the kind Docker network’s real subnet, which kind bumps off 172.18 when older networks hold it. Off-subnet pools do not error; the request through the front door just times out.

Modelplane: Pros and Cons#

Pros#

Cons#

When to use Modelplane#

Use it when:#

• You already run, or are about to run, three or more GPU clusters across regions, clouds or a neocloud, and the matchmaking job is eating a platform engineer’s week
• You want ML teams to ask for hardware without learning infrastructure, and you want that contract to be one you did not have to invent
• Crossplane v2 is already in your estate, or you are willing to adopt it as the control-plane engine
• You are evaluating DRA as a platform contract, not just as the device-plugin replacement, and want a worked example of the pattern

Consider alternatives when:#

• You have one cluster. KServe or llm-d serve models; Modelplane places them. With one cluster there is nothing to place.
• You need production stability today. This is v1alpha1, and the release process has sharp edges.
• Your GPU clusters are shared with other workloads. The capacity ledger assumes exclusivity.
• You cannot run Crossplane, or a second reconciliation engine is a non-starter for your ops team.

Troubleshooting#

Hands-On Demo Repository#

Everything above is reproducible from srekubecraft-demo/modelplane-fleet:

modelplane-fleet/
  Taskfile.yml
  kubernetes/
    clusters/            # three kind configs, node image pinned by digest
    modelplane/
      configuration.yaml          # the package pin the docs get wrong
      10-inference-gateway.yaml   # Traefik front door, MetalLB pool templated
      20-inference-class.yaml     # platform side: the device vocabulary
      30-inference-clusters.yaml  # two BYO clusters, region-labelled
      40-model-deployment.yaml    # developer side: CEL over that vocabulary
      50-model-service.yaml       # one endpoint in front of every replica

task up          # clusters -> workloads -> control plane -> register -> deploy
task placement   # where did each replica land?
task curl        # one request through the single front door
task drain CLUSTER=gpu-eu-west
task undrain CLUSTER=gpu-eu-west
task down        # delete all three clusters

The bootstrap RBAC, the cloud-provider trim and the fake DRA driver are pulled from upstream at the pinned tag rather than vendored. Budget 24 GB of Docker memory and about 40 minutes for the first task up, most of it the two serving stacks.

Conclusion#

Modelplane is the first thing I have run that treats the fleet as the unit, and the part I would keep even if the project vanished is the contract mechanism. Publishing a device vocabulary in DRA’s own terms, letting developers write DRA-shaped CEL against it, and leaving real DRA admission authoritative on the workload cluster: that pattern removes the translation table every GPU platform team ends up maintaining, and it generalises to any hardware a driver can describe. The claim: Synthetic flag is the pragmatic touch that makes it shippable while drivers catch up.

The scheduling model is deliberately dumb in the right ways. Existing replicas are inputs, not decisions, so nothing churns. Capacity is charged in nodes, so it under-counts rather than overcommits. Drain is a taint. Every one of those choices showed up in the demo exactly as the docs describe, which is more than I can say for most v1alpha1 projects.

What has not caught up is the release engineering. A tag whose install manifest points at the previous, broken release is the kind of thing that costs an evaluator a day, and I spent that day. Add the kubelet 1.34.0 DRA deadlock, which is Kubernetes’ bug but lands on whoever pins that image, and the honest advice is: evaluate it now, on the laptop path, for the design ideas; do not put production traffic behind it yet.

If you have one GPU cluster, stay on KServe or llm-d. If you have three, in three places, and the question “which model runs where” is a ticket queue, this is the shape of the answer.

If you found this useful, you might also enjoy my related posts on AI infrastructure and platform design:

• llm-d - Distributed Inference on Kubernetes
• KServe - Model Serving on Kubernetes
• Kratix - Building Platforms as a Product
• Introduction to Crossplane