Two schedulers, one fabric: Slurm marshals gang-scheduled training across thousands of accelerators while Kubernetes drives elastic inference. Both run inside the nation's borders, under the operator's keys, on hardware the operator owns.
A single scheduler cannot serve both tightly-coupled training jobs and bursty, latency-sensitive inference — the orchestration layer runs each on the engine built for it.
Batch training runs on Slurm, which gang-schedules whole job steps so every rank in a distributed run starts together or not at all. Partitions, QoS tiers, and reservations map directly onto how a sovereign operator allocates compute across national programs.
Model-serving and specialized-model endpoints run on Kubernetes, where horizontal autoscaling and rolling updates keep inference available while individual pods are replaced. Inference services scale against real request load rather than static reservations.
Both planes draw from one accounting of GPUs, interconnect, and power, so a node released from a completed training job can be returned to the inference pool without manual reconciliation. There is a single source of truth for what is allocated and what is idle.
The scheduler control plane, its state store, and its accounting database run on operator-owned infrastructure inside the data residency boundary. No workload placement decision or job metadata leaves the nation.
Collective operations across hundreds of ranks are bounded by the slowest link, so the scheduler is topology-aware down to the switch and rail.
A training step is admitted only when every requested accelerator is simultaneously available, preventing partial allocations that would deadlock a collective. Jobs queue as an atomic unit and are never left holding a fraction of what they need.
The scheduler places co-dependent ranks to minimize hops across the interconnect, preferring nodes on the same rail and leaf before spanning spines. Placement is fed by the actual wiring of the cluster, not an abstract node count.
Smaller jobs backfill idle windows ahead of large reservations only when they provably finish before the reserved start time. Throughput rises without pushing large national training runs indefinitely down the queue.
Named reservations hold capacity for time-bound obligations — a scheduled foundation-model run or a regulator-facing evaluation — with hard start and end boundaries. Capacity is committed by policy, not by whoever submits first.
When a higher-priority sovereign workload arrives, lower-priority work yields cleanly rather than being killed and lost.
Jobs carry a priority derived from their QoS tier and the submitting tenant's allocation, and the scheduler preempts strictly down that ordering. The rules are declared in advance and applied uniformly, not negotiated per incident.
Preempted training jobs receive a grace signal to flush a checkpoint before eviction, so a run resumes from its last consistent state instead of restarting. The scheduler cooperates with the training loop rather than terminating it abruptly.
Evicted jobs are automatically requeued with their reservation and dependency graph intact, and resume when priority space reopens. Operators do not resubmit by hand after a preemption event.
Latency-sensitive inference is isolated from preemptible batch training so a burst of training never degrades a live model endpoint. The two workload classes cannot starve each other.
Scheduling is not just placement — it is an auditable record of who ran what, on which hardware, under which allocation.
Submissions pass through admission control that validates the request against the tenant's quota, residency constraints, and image provenance before it enters any queue. A job that violates policy is rejected at the door, not discovered mid-run.
Accelerator-seconds, energy, and interconnect use are attributed to the submitting tenant and program, producing a defensible record of national compute consumption. Usage is measured against declared allocations rather than estimated after the fact.
Node or link failures during a run are detected, the affected ranks drained, and the job restarted from checkpoint on healthy hardware. Long training runs survive the routine failures inherent to hyperscale clusters.
Job lifecycle events — submit, admit, preempt, complete — are recorded in a tamper-evident log so the history of what ran on sovereign hardware is verifiable. The scheduling record carries the same evidentiary weight as the settlement ledger.