Kubernetes 1.37 Gang Scheduling - The Workload API, a Silently Dropped Field, and Where Preemption Stops

You give a four-rank training job a cluster with six GPUs. The scheduler places three ranks, runs out of devices, and leaves the fourth Pending. A second four-rank job arrives and takes the other three. Now six GPUs are allocated, two pods are Pending forever, and zero jobs are training. Nothing is broken. Every pod that could be scheduled was scheduled. That is exactly the problem.

This is partial-placement deadlock, and until recently Kubernetes had no answer for it in tree. You reached for Volcano, or Kueue, or the coscheduling plugin, and you accepted a second scheduler or a controller that queued work outside the API you already had.

KEP-4671 changes that. Kubernetes v1.37, released 2026-08-26, promotes the Workload and PodGroup APIs to beta along with workload-aware preemption. All-or-nothing placement is now a field on a Job.

It is also off by default, and the ergonomic half of it is alpha and fails without telling you. I built the deadlock on a laptop, fixed it with one field, and then spent longer than I want to admit finding out why the field did nothing the first time.

Provenance: every gate default, API shape and milestone below is read from kubernetes/kubernetes at release-1.37 and from the KEP text, cited with file:line. Every command output is from a four-node Kind cluster running kindest/node:v1.37.0 with dra-example-driver v0.5.0 advertising two fake GPUs per node. Where I did not verify something, I say so.

Who Should Read This?#

This post is for:

• Platform Engineers running batch or training workloads who have been carrying Volcano or Kueue purely for gang semantics and want to know whether in-tree is ready
• SREs who own a shared GPU pool and have watched accelerators sit allocated and idle behind a job that will never reach quorum
• Anyone planning a 1.36 to 1.37 upgrade with the alpha gang scheduling gates already enabled, because two of them changed name or shape in a way that will break your config
• Anyone building on DRA, since the gang scheduling story and the DRA story intersect in exactly one place, and it is a gap

TL;DR#

• Gang scheduling is beta in 1.37 and disabled by default. One gate, GenericWorkload, now covers the Workload API, gang scheduling and workload-aware preemption. It goes on kube-apiserver, kube-scheduler and kube-controller-manager.
• The gates GangScheduling and WorkloadAwarePreemption were merged into GenericWorkload in 1.37. If they are in your config, remove them before you upgrade.
• The field you actually want, Job.spec.scheduling.schedulingPolicy.gang.minCount, needs a second gate, WorkloadWithJob, which is alpha and off. Without it the field is pruned silently: kubectl apply reports success, no warning, no event, and nothing changes.
• Workload-aware preemption does not reclaim DRA devices in 1.37. You can gang-schedule GPUs allocated through DRA. You cannot gang-preempt them. Node-local extended resources preempt fine.
• The default disruptionMode is single. Gang is all-or-nothing at admission, not at eviction. A higher-priority gang will break a running gang pod by pod unless you set disruptionMode: all.
• Three new scheduler metrics ship with beta, all prefixed scheduler_podgroup_.

What KEP-4671 actually shipped#

Two objects and one pod field.

The split matters. Workload is the shape you declare once; PodGroup is the instance the scheduler admits or rejects as a unit. From staging/src/k8s.io/api/scheduling/v1beta1/types.go, the policy itself is as small as it gets:

type PodGroupSchedulingPolicy struct {
	Basic *BasicSchedulingPolicy `json:"basic,omitempty"`
	Gang  *GangSchedulingPolicy  `json:"gang,omitempty"`
}

type GangSchedulingPolicy struct {
	MinCount int32 `json:"minCount"`
}

That is the whole feature. minCount pods are placed together or none are.

The pod side is equally small, from staging/src/k8s.io/api/core/v1/types.go:4885:

type PodSchedulingGroup struct {
	PodGroupName *string `json:"podGroupName,omitempty"`
}

The flow#

flowchart TB
    subgraph User["What you write"]
        J["Job
        spec.scheduling.schedulingPolicy.gang.minCount: 4"]
    end

    subgraph Controllers["kube-controller-manager"]
        W["Workload
        podGroupTemplates[0]"]
        PG["PodGroup
        schedulingPolicy.gang.minCount: 4"]
    end

    subgraph Pods["Pods"]
        P0["rank 0
        spec.schedulingGroup.podGroupName"]
        P1["rank 1"]
        P2["rank 2"]
        P3["rank 3"]
    end

    subgraph Sched["kube-scheduler"]
        D{"Can all 4
        be placed?"}
        Y["Bind all 4"]
        N["Bind none
        PodGroup Unschedulable"]
    end

    J --> W --> PG
    J --> P0 & P1 & P2 & P3
    P0 & P1 & P2 & P3 --> D
    PG --> D
    D -->|yes| Y
    D -->|no| N

The feature gates, and the one that will waste your afternoon#

This is the part the release notes gloss over. Every gate in this area is off by default in 1.37, and they are not independent. Read from pkg/features/kube_features.go at release-1.37:

GenericWorkload gets you the API. It does not get you the Job field.

WorkloadWithJob is KEP-5547, described in the source as enabling “the Job controller to automatically create Workload and PodGroup objects for Jobs that qualify for gang scheduling”. It is alpha, it is off, and without it the Job field does not error. It disappears.

$ kubectl apply -f 20-gang.yaml
job.batch/train-a created
job.batch/train-b created

$ kubectl -n ml-team get job train-a -o jsonpath='{.spec.scheduling}'
                                      # nothing. the field is gone.

No warning, no event, no condition. The Job runs with the old pod-by-pod behaviour and you go looking for a scheduler bug that does not exist.

Why This Matters: a beta feature whose headline ergonomics sit behind an alpha gate is a feature most people will try once, conclude is broken, and walk away from. If you evaluate this in 1.37, enable both gates or you are not evaluating it at all.

Upgrading from 1.36#

Two things will bite, both from the KEP’s own upgrade notes (README.md:1564-1572):

1. GangScheduling no longer exists. It was merged into GenericWorkload, along with WorkloadAwarePreemption from KEP-5710. Remove it from your feature gate config before upgrading. On a downgrade to 1.36 you have to re-enable it by hand.
2. scheduling.k8s.io/v1alpha2 is gone, replaced by v1alpha3. Delete every v1alpha2 resource before you upgrade. There is no backward conversion from v1alpha3.

Why not Kueue, Volcano, or the coscheduling plugin?#

The KEP explicitly replaces KEP-583 (coscheduling) and KEP-5832, per kep.yaml. The coscheduling plugin is the thing being retired here.

Kueue is not being retired, and the KEP is direct about that: the near-term goal is for Kueue to be aware of workload-aware scheduling, and the long-term goal is for Kueue to use it as its engine. Kueue gives you queues, quota and fair sharing. KEP-4671 gives you placement atomicity. They are different layers.

Bottom line: if all you needed from Volcano was gang semantics, 1.37 lets you delete it. If you needed quota and fair sharing, you still want Kueue, and it will sit on top of this rather than beside it.

What this is NOT#

• Not a queueing system. There is no quota, no fair share, no borrowing, no priority-based admission beyond preemption. A rejected PodGroup retries; it does not wait in a queue with siblings.
• Not topology-aware yet. TopologyAwareWorkloadScheduling is a separate alpha gate. Placing a gang together is not the same as placing it on one rack.
• Not a DRA preemption story. Covered below, and it is the biggest gap.
• Not on by default. Worth repeating, because “beta” usually implies otherwise.

The demo: six fake GPUs, no hardware#

Four Kind nodes, dra-example-driver v0.5.0 with numDevices: 2, so three workers advertise two fake GPUs each. Six devices total. Two jobs, four ranks each, one GPU per rank. Demand eight, supply six.

The cluster config carries all three gates:

# kubernetes/cluster/kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: gang-demo

featureGates:
  # API + gang + workload-aware preemption
  GenericWorkload: true
  # without it spec.scheduling is pruned silently
  WorkloadWithJob: true
  # shared claims per PodGroup
  DRAWorkloadResourceClaims: true

runtimeConfig:
  "scheduling.k8s.io/v1beta1": "true"

nodes:
  - role: control-plane
    image: kindest/node:v1.37.0
  - role: worker
    image: kindest/node:v1.37.0
  - role: worker
    image: kindest/node:v1.37.0
  - role: worker
    image: kindest/node:v1.37.0

Verify the API is actually served before anything else. If this is empty, everything after it fails quietly:

$ kubectl api-resources --api-group=scheduling.k8s.io
NAME              SHORTNAMES   APIVERSION                  NAMESPACED   KIND
podgroups                      scheduling.k8s.io/v1beta1   true         PodGroup
priorityclasses   pc           scheduling.k8s.io/v1        false        PriorityClass
workloads                      scheduling.k8s.io/v1beta1   true         Workload

$ kubectl get resourceslices \
    -o jsonpath='{range .items[*]}{.spec.nodeName}{": "}{range .spec.devices[*]}{.name}{" "}{end}{"\n"}{end}'
gang-demo-worker: gpu-0 gpu-1
gang-demo-worker2: gpu-0 gpu-1
gang-demo-worker3: gpu-0 gpu-1

Example 1: the deadlock#

Two ordinary Jobs, no scheduling policy. Each rank takes one GPU through a ResourceClaimTemplate:

# kubernetes/workloads/10-no-gang.yaml
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: one-gpu
  namespace: ml-team
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.example.com
            allocationMode: ExactCount
            count: 1
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-a
  namespace: ml-team
spec:
  completions: 4
  parallelism: 4
  backoffLimit: 0
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
        - name: gpu
          resourceClaimTemplateName: one-gpu
      containers:
        - name: rank
          image: registry.k8s.io/e2e-test-images/busybox@sha256:0ffbe172f8d245c83f285c6992b452c53d085661e03ddfd3b484332026e6c8bb
          command: ["sh", "-c", "echo rank $JOB_COMPLETION_INDEX holding a GPU; sleep 3600"]
          resources:
            requests: { cpu: 50m, memory: 32Mi }
            limits: { cpu: 100m, memory: 64Mi }
            claims:
              - name: gpu

Applied, with train-b identical:

$ kubectl -n ml-team get pods \
    -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME              STATUS    NODE
train-a-0-g5dg8   Running   gang-demo-worker
train-a-1-wmbm2   Running   gang-demo-worker2
train-a-2-6gxvp   Running   gang-demo-worker
train-a-3-fn2n9   Pending   <none>
train-b-0-ccwqd   Running   gang-demo-worker3
train-b-1-z2rjp   Running   gang-demo-worker3
train-b-2-s67tv   Running   gang-demo-worker2
train-b-3-kdwvr   Pending   <none>

Three and three. Six GPUs allocated, six containers running, and not one of them can do a single all-reduce, because neither job has four ranks. The scheduler is not confused:

FailedScheduling: 0/4 nodes are available: 1 node(s) had untolerated taint(s),
3 cannot allocate all claims. preemption: 0/4 nodes are available:
4 Preemption is not helpful for scheduling.

Result: 6/6 GPUs consumed, 0 jobs able to train, and no error anywhere that tells you so.

One honest caveat: this split is not deterministic. On a later run of the same manifests I got train-a with all four ranks and train-b with two running and two Pending, which is one job training and one wedged. Without gang semantics the outcome depends on pod arrival order. Sometimes a job gets lucky. That is the actual argument for the feature: it replaces luck with a guarantee.

Example 2: the one field#

Same two jobs. One block added:

spec:
  completions: 4
  parallelism: 4
  scheduling:
    schedulingPolicy:
      gang:
        minCount: 4

$ kubectl -n ml-team get podgroups
NAME                                         POLICY   WORKLOAD             STATUS          AGE
train-a-88856b5bc-train-a-pgt-0-cc865b9db    Gang     train-a-88856b5bc    Scheduled       93s
train-b-66c995d6cb-train-b-pgt-0-fc65464f8   Gang     train-b-66c995d6cb   Unschedulable   93s

$ kubectl -n ml-team get pods --no-headers \
    -o 'custom-columns=N:.metadata.labels.app\.kubernetes\.io/name,S:.status.phase' \
  | sort | uniq -c
   4 train-a   Running
   4 train-b   Pending

The Job controller created a Workload and a PodGroup per job, wired every pod to its group, and the scheduler admitted one gang and rejected the other whole:

PodGroupInitiallyScheduled=False  Unschedulable
  minCount (4) cannot be satisfied: 2 scheduled, 1 remaining,
  pod group preemption: No preemption victims found for incoming preemptor

Result: four GPUs doing real work, two free for whatever comes next, and a PodGroup whose status says plainly why the second job is waiting.

Note: the counter in that message did not line up with minCount in any of my runs (“2 scheduled, 1 remaining” against a minCount of 4). The scheduling behaviour was correct every time. I did not chase the message arithmetic, so treat the prose as informative and the condition status as authoritative.

Preemption, and the two places it stops#

This is where I lost the afternoon, and where the post earns its keep.

It does not reclaim DRA devices#

With train-a holding four GPUs at priority 100 and two GPUs free, I added a third gang at priority 10000. It should preempt. It does not:

$ kubectl -n ml-team get podgroup \
    -o jsonpath='{range .items[*]}{.metadata.name}{" priority="}{.spec.priority}{" "}{range .status.conditions[*]}{.reason}{end}{"\n"}{end}'
train-a-...      priority=100     Scheduled
train-b-...      priority=100     Unschedulable
train-urgent-... priority=10000   Unschedulable

Priority propagated correctly. Gang preemption ran. It found no victims. The reason is in KEP-5710, in a list of known gaps:

Fixing global constraint tracking for nominations will be pursued as a separate enhancement, most likely before KEP-5690: DRA Workload Resource Claims starts supporting Workload-Aware Preemption (since DRA resources are often not node-local).

Workload-aware preemption is cluster-scoped, but nominations are still tracked per node. DRA devices are frequently not node-local. So DRA support for workload-aware preemption is future work, tracked under KEP-5690.

To confirm the limit is DRA-specific and not preemption being broken, I advertised a plain node-local extended resource instead:

$ kubectl patch node gang-demo-worker --subresource=status --type=json \
    -p '[{"op":"add","path":"/status/capacity/example.com~1accel","value":"2"}]'

Same two gangs, same priorities, example.com/accel: 1 per rank instead of a ResourceClaim. Preemption fires immediately:

accel-research-0-thdnd   Preempted by podgroup 11a0e2df-a4c4-4c67-a0dc-04a479f774ba on node cluster
accel-research-3-dn4gv   Preempted by podgroup 11a0e2df-a4c4-4c67-a0dc-04a479f774ba on node cluster

Note on node cluster. That is the cluster-wide scope working as designed.

Result: in 1.37 you can gang-schedule DRA GPUs. You cannot gang-preempt them. If your GPU pool depends on preemption to give production work priority over research work, this feature does not yet cover you.

Gang is atomic at admission, not at eviction#

Look again at that output. Two pods preempted, out of a gang of four. The other two kept running, holding accelerators, unable to reach quorum. The exact deadlock the feature exists to prevent, arrived at from the other direction.

The reason is a default:

$ kubectl -n ml-team get podgroups -o custom-columns=NAME:.metadata.name,DISRUPT:.spec.disruptionMode
NAME                       DISRUPT
accel-research-...         map[single:map[]]

disruptionMode defaults to single. The fix is one more block on the Job:

  scheduling:
    schedulingPolicy:
      gang:
        minCount: 4
    # evict the gang as a unit
    disruptionMode:
      all: {}

Same scenario with all, counting the Preempted events for the research gang’s pods:

accel-research-0-xrh27
accel-research-1-d4k2w
accel-research-2-rvs82
accel-research-3-849fs

All four, as a unit.

Tip: treat disruptionMode: all as part of the gang configuration, not an optional extra. minCount without it buys you an atomic start and a non-atomic death, which for a distributed training job is most of the problem back.

One more thing worth knowing: the whole spec.scheduling block is immutable after creation. Only schedulingPolicy.gang.minCount can change. Getting it wrong means recreating the Job:

The Job "accel-research" is invalid: spec.scheduling.disruptionMode:
Invalid value: null: field is immutable

Metrics#

Three metrics ship with the beta promotion, listed in kep.yaml. The scheduler image is distroless, so scrape it over a port-forward with the admin client certificate:

$ curl -sk --cert c.crt --key c.key https://127.0.0.1:10259/metrics | grep '^scheduler_podgroup'
scheduler_podgroup_schedule_attempts_total{profile="default-scheduler",result="scheduled"} 2
scheduler_podgroup_schedule_attempts_total{profile="default-scheduler",result="unschedulable"} 22
scheduler_podgroup_scheduling_algorithm_duration_seconds_count 24
scheduler_podgroup_scheduling_algorithm_duration_seconds_sum 0.039103

The unschedulable counter is the one to alert on. A gang that cannot be placed retries continuously, so a climbing result="unschedulable" with a flat result="scheduled" is a pool that is full or a minCount nobody can satisfy.

Pros and Cons#

Pros#

Cons#

When to use it, and when not#

Use it when:

• You run distributed training or MPI-style jobs where partial placement is worthless
• You already have kube-scheduler and want to drop Volcano, carrying it only for gang semantics
• Your scarce resource is node-local: CPU, memory, or an extended resource like a device-plugin GPU
• You are on 1.37+ and can set feature gates on the control plane, which rules out most managed control planes today

Do not use it when:

• You need quota, fair sharing or borrowing across teams. Use Kueue
• Your GPUs come through DRA and you depend on preemption. Wait for KEP-5690
• You need topology-aware placement now. That is a separate alpha
• You cannot set feature gates on kube-apiserver and kube-controller-manager

Troubleshooting#

Hands-On Demo Repository#

Everything above is reproducible on a laptop, no GPU required. The demo lives in the srekubecraft-demo monorepo under gang-scheduling/:

• kubernetes/cluster/kind.yaml - four nodes on v1.37.0 with all three gates
• kubernetes/cluster/dra-example-driver.yaml - upstream chart v0.5.0 rendered with two fake GPUs per node
• kubernetes/workloads/ - the deadlock, the gang fix, the DRA preemption gap, and both disruptionMode variants
• Taskfile.yml - one task per act

git clone https://github.com/nicknikolakakis/srekubecraft-demo
cd srekubecraft-demo/gang-scheduling

task up          # cluster, DRA driver, extended resource, namespace, priority classes
task deadlock    # two jobs, six GPUs, nothing trains
task gang        # the same jobs with gang.minCount
task dra-preempt # the DRA preemption gap
task preempt     # node-local preemption, disruptionMode single
task preempt:all # the same, evicted as a unit
task metrics     # the three PodGroup metrics
task down        # delete the cluster

Conclusion#

Gang scheduling in tree is the right call. The coscheduling plugin was always a workaround, Volcano is a lot of machinery to carry for one semantic, and minCount on a Job is the API this should have had years ago. When it works, it works cleanly: one gang trains, the other waits, and the PodGroup status tells you which and why.

The catch is that 1.37 is not where you adopt it. Three things stand in the way. The Job field is alpha and fails silently, which will burn anyone evaluating this in an afternoon. The disruptionMode default undoes half the guarantee under preemption. And workload-aware preemption cannot reclaim DRA devices, which matters because DRA is how modern GPU scheduling works and preemption is how you stop research jobs from starving production ones.

Practical advice, in order:

• On 1.37, turn both gates on in a test cluster and run your real job shapes. The behaviour is solid. The packaging is not.
• Set disruptionMode: all from the first manifest. It is immutable, and the default is wrong for anything that deserves a gang.
• If your GPUs come through DRA, keep Volcano or Kueue for now. Scheduling is covered; preemption is not.
• If your scarce resource is node-local, this is already enough to delete a scheduler.
• Watch 1.38. kep.yaml targets stable, which is when the gates default on and this stops being an opt-in.

I will revisit this when 1.38 ships, with the same demo, to see whether KEP-5690 has closed the DRA preemption gap.

If you found this useful, you might also enjoy my related posts:

• Modelplane - Fleet Inference Across Clusters, Demoed Without a Single GPU
• Modelplane Beyond Placement - Routing Policy, the Dynamo Stack, and Four Gotchas
• llm-d - Kubernetes-Native Distributed LLM Inference at Scale
• KServe - Production ML Serving on Kubernetes, from sklearn to LLMs
• Kubernetes Autoscaling Beyond HPA - Event-Driven KEDA and VPA Resource Optimization