Karmada v1.19 Multi-Component Scheduling - One Training Job, Three Clusters, Four Silent Failures

You have three GPU clusters and one control plane in front of them. Someone submits a distributed training job: one master, two workers. You want the whole job in one cluster, because NCCL across regions is not a thing you do on purpose. You write the obvious policy: divide the replicas, pack them into as few clusters as possible.

The job runs three times. Once in every cluster. Every status in the control plane is green.

That was my first run of Karmada v1.19 against a PyTorchJob. Karmada graduated from the CNCF in September 2026, and the headline of v1.19 is exactly this use case. The feature works. It needs four things to be true at the same time, and when any of them is missing, nothing tells you.

Provenance: code references are from karmada-io/karmada at tag v1.19.0. Every output is from four kind clusters (kindest/node:v1.37.0) on a laptop, one Karmada host and three members with fake nvidia.com/gpu capacity.

TL;DR#

• MultiplePodTemplatesScheduling is alpha and off by default. It goes on karmada-controller-manager, karmada-scheduler and karmada-webhook.
• Gate off: a PyTorchJob has no replica information, Karmada treats it like a ConfigMap and copies it to every cluster.
• Gate on, no spread constraint: still copied to every cluster. The new path only runs with spreadByField: cluster, minGroups: 1, maxGroups: 1.
• With the constraint, every GPU job fails with no enough resource. The default resource models only know CPU and memory. Turn CustomizedClusterResourceModeling off on the scheduler.
• The general estimator sums cluster totals. A job can land where no node fits its workers, report Scheduled and FullyApplied, and stay Pending forever. Deploy the accurate scheduler-estimator.

What v1.19 actually shipped#

The multiple pod templates proposal teaches the interpreter, the scheduler and the estimators about workloads with more than one pod template. The interpreter returns one component per template, the scheduler places them as one unit in one cluster, and v1.19 records the per-component result on the ResourceBinding as spec.clusters[*].components.

Built-in interpreters ship for Kubeflow PyTorchJob, TFJob and MPIJob, RayCluster, RayJob, Volcano Job, FlinkDeployment and SparkApplication. The Kubeflow ones are the kubeflow.org/v1 training-operator APIs. Kubeflow Trainer v2’s TrainJob has no built-in interpreter.

Why not MultiKueue?#

MultiKueue solves the same problem from the queueing side. I did not run it, the right column is from its docs.

Bottom line: if you already run Karmada for services, v1.19 is the first release where training on the same control plane is realistic. If you only need batch dispatch, MultiKueue gets there with a beta feature.

What it is NOT#

• Not gang scheduling inside the member. That is kube-scheduler’s job, see Kubernetes 1.37 Gang Scheduling
• Not job splitting. A multi-component job never spans clusters
• Not rescheduling. v1.19 is “laying the foundation” for it. A misplaced job stays put

The demo#

flowchart TB
    subgraph Host["kind: karmada-host"]
        API["karmada-apiserver"]
        SCH["karmada-scheduler"]
        EST["scheduler-estimator x3"]
    end
    subgraph A["member-a"]
        A1["1 node, 2 GPU"]
    end
    subgraph B["member-b"]
        B1["3 nodes, 2 GPU each"]
    end
    subgraph C["member-c"]
        C1["1 node, 8 GPU"]
    end
    API --> SCH
    SCH --> A & B & C
    EST -. per-node view .-> A & B & C

The GPUs are a node status patch. Karmada sums node allocatable into each cluster’s resource summary, and kube-scheduler in each member enforces it per node:

$ kubectl --kubeconfig .karmada/karmada-apiserver.config get clusters -o custom-columns=...
NAME       ENDPOINT                              GPU   CPU
member-a   https://member-a-control-plane:6443   2     12
member-b   https://member-b-control-plane:6443   6     48
member-c   https://member-c-control-plane:6443   8     12

Each member runs the Kubeflow training-operator v1.9.4, so the jobs become real pods running pause. train-small is a CPU master plus two 1-GPU workers. train-big is the same with 3 GPUs per worker:

    Worker:
      replicas: 2
      template:
        spec:
          containers:
            - name: pytorch
              image: registry.k8s.io/pause:3.10.1
              resources:
                requests: {cpu: 50m, memory: 32Mi, nvidia.com/gpu: "3"}
                limits: {cpu: 100m, memory: 64Mi, nvidia.com/gpu: "3"}

Failure 1: gate off, a PyTorchJob is a ConfigMap#

The policy most people write for a training job:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: training
  namespace: ml-team
spec:
  resourceSelectors:
    - apiVersion: kubeflow.org/v1
      kind: PyTorchJob
  placement:
    clusterAffinity:
      clusterNames: [member-a, member-b, member-c]
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Aggregated

On a fresh karmadactl init:

replicas: null
components: null
Binding has been scheduled successfully. Result: {member-a:0, member-b:0, member-c:0}

-- member-a
pod/train-small-master-0   1/1   Running
pod/train-small-worker-0   1/1   Running
pod/train-small-worker-1   1/1   Running
-- member-b  (same three pods)
-- member-c  (same three pods)

Zero replicas assigned, applied everywhere. The built-in interpreter defines componentResource but no replicaResource. With the gate off, Karmada falls back to the replica hook (pkg/detector/detector.go:1513), finds nothing, and the scheduler treats a binding with no replicas and no components as a non-workload (pkg/scheduler/core/util.go:66-69). Non-workloads go to every cluster. Divided and Aggregated never get a vote.

Result: with the gate off, a Karmada policy for Kubeflow v1 jobs is a broadcast.

Failure 2: gate on, still three copies#

Enable the gate on all three components and run the same policy:

NAME                     REPLICAS   COMPONENTS      CLUSTERS
train-small-pytorchjob   <none>     master,worker   member-b,member-a,member-c

Same nine pods. Karmada now knows the components, and nothing uses them. The multi-component path is behind one check (pkg/scheduler/core/estimation.go:43-65):

for i := range spec.Placement.SpreadConstraints {
	if spec.Placement.SpreadConstraints[i].SpreadByField == policyv1alpha1.SpreadByFieldCluster &&
		spec.Placement.SpreadConstraints[i].MinGroups == 1 &&
		spec.Placement.SpreadConstraints[i].MaxGroups == 1 {
		return true
	}
}
return false

Without that exact constraint, a multi-component workload is propagated “to all candidate clusters” (pkg/scheduler/core/common.go:72).

Result: replicaDivisionPreference: Aggregated reads like “one cluster if possible”. For multi-component workloads it means nothing.

Failure 3: 16 free GPUs, and no room#

Add the constraint:

    spreadConstraints:
      - spreadByField: cluster
        minGroups: 1
        maxGroups: 1

==> Scheduled:
False failed to select clusters: no enough resource when selecting 1 clusters

Two GPUs requested, sixteen free. The same job without the GPU request schedules immediately, into one cluster, with the per-component result recorded. So the path works and something in it cannot count GPUs.

That something is cluster resource modeling: beta, on by default since v1.4. The general estimator simulates the job against fake nodes built from each cluster’s resource models (pkg/estimator/client/general.go:241), and the default models only have two dimensions:

- grade: 0
  ranges:
    - {name: cpu, min: "0", max: "1"}
    - {name: memory, min: "0", max: 4Gi}
# ... up to grade 8, cpu and memory only

No GPU on any simulated node, so every GPU job fits zero times. Custom models cannot fix it, the docs allow only “cpu, memory, storage, ephemeral-storage”. The fix is turning CustomizedClusterResourceModeling off on the scheduler:

NAME                     COMPONENTS      CLUSTERS   PER-CLUSTER
train-small-pytorchjob   master,worker   member-c   [map[name:master replicas:1] map[name:worker replicas:2]]

member-c  train-small-master-0   Running
member-c  train-small-worker-0   Running
member-c  train-small-worker-1   Running

The whole job in one cluster. This is the v1.19 feature working.

Result: with default settings, multi-component scheduling cannot place a GPU job, and the error points at capacity. CPU-only demos will not show you this.

Failure 4: Scheduled, FullyApplied, Pending forever#

train-big needs two workers with 3 GPUs each. member-b has six GPUs, but two per node:

cond:
  - Scheduled=True Binding has been scheduled successfully.
  - FullyApplied=True All works have been successfully applied
clusters:
  - member-b

member-b  train-big-master-0   Running
member-b  train-big-worker-0   Pending
member-b  train-big-worker-1   Pending
# 0/4 nodes are available: ... 3 Insufficient nvidia.com/gpu

Green in Karmada, stuck forever in the member, while member-c has eight free GPUs on one node. The general estimator divides cluster totals by the job’s request, and in its own words “has no per-node view” (pkg/estimator/client/general.go:176).

The fix is the accurate scheduler-estimator, one per member, which answers from real nodes:

clusters:
  - member-c

member-c  train-big-master-0   Running
member-c  train-big-worker-0   Running
member-c  train-big-worker-1   Running

It does not rescue you from Failure 3. The scheduler takes the minimum across estimators, and with modeling back on the general estimator’s zero wins again: no enough resource when selecting 1 clusters.

Result: for GPU jobs you need both: modeling off, and the accurate estimator.

The working configuration#

Pros and Cons#

Hands-On Demo Repository#

Everything above is in srekubecraft-demo/karmada-multi-component, with one task per failure:

task up                              # 4 kind clusters, fake GPUs, Karmada, training-operator
task copies                          # Failure 1
task gate:on && task copies:gated    # Failure 2
task one                             # Failure 3
task modeling:off && task one        # fixed
task fragment                        # Failure 4
task estimator && task fragment      # fixed

The README also covers the setup traps of running Karmada on kind under macOS, including karmadactl init and join recording endpoints the host cluster cannot reach.

Conclusion#

Karmada v1.19 does what the release notes say. I watched it put a PyTorchJob into one cluster and record which component got how many replicas. Getting there took four changes, and each missing one looked like success or like a capacity problem. Use the configuration table above, and do not alert on Karmada conditions alone: Scheduled=True and FullyApplied=True were both true for a job whose workers could never start.

The feature is alpha, and its defaults were built for stateless replicas while the GPU case is the reason it exists. Run it in a staging fleet with the configuration above, and watch the rescheduling work in the next releases.

If you found this useful, you might also enjoy my related posts on scheduling and multi-cluster AI workloads:

• Kubernetes 1.37 Gang Scheduling - The Workload API, a Silently Dropped Field, and Where Preemption Stops
• Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU
• llm-d - Kubernetes-Native Distributed LLM Inference at Scale