Training a sovereign foundation model is a synchronization problem before it is a compute problem. The fabric that carries gradients between thousands of GPUs decides whether the model finishes on schedule or stalls waiting on the wire.
At multi-node scale the bottleneck moves off the GPU and onto the links that connect them.
Every training step ends with an all-reduce over model gradients across all participating GPUs. On large clusters this collective can consume a significant fraction of wall-clock time, so link bandwidth and latency directly govern tokens-per-second.
Synchronous training advances at the pace of the slowest link in the collective. A single congested or lossy path idles thousands of accelerators, which is why the fabric is engineered for the worst case, not the average.
Both InfiniBand and RoCE are built to run without packet loss under load. Dropped packets trigger retransmission and collapse collective throughput, so the transport is designed to never rely on drop-and-retry as a congestion signal.
We treat InfiniBand and RoCE as interchangeable implementations of the same guarantee: RDMA transport, lossless delivery, and hardware offload. The choice is an operational and sovereignty decision, not a change in application behavior.
InfiniBand is engineered end to end for RDMA rather than adapted from Ethernet.
Link-level credits mean a sender only transmits when the receiver has advertised buffer space. Congestion is prevented at the source rather than absorbed by dropping frames, giving deterministic, lossless behavior under sustained collective load.
A centralized subnet manager computes forwarding tables for the entire fabric and programs deterministic routes. Paths are assigned globally with topology awareness rather than discovered hop-by-hop, which is what makes rail-optimized layouts predictable.
Switch-resident reduction offload can aggregate gradient data inside the network as it converges toward the root, cutting the volume that crosses the fabric and shortening the all-reduce for large collectives.
The fabric can steer flows around hotspots in hardware, spreading traffic across parallel paths in a fat-tree so a single oversubscribed link does not throttle an entire collective.
RoCEv2 delivers the same RDMA semantics on routable Ethernet, using standard tooling and staff.
RoCEv2 encapsulates RDMA in UDP over IP, so it routes across standard Layer 3 boundaries. Operators keep familiar Ethernet management while GPUs still bypass the kernel and talk memory-to-memory.
Priority Flow Control pauses a specific traffic class at the link when buffers fill, giving RDMA a lossless lane over Ethernet. It is tuned per class so control and storage traffic are not starved by RDMA pauses.
Explicit Congestion Notification marks packets as queues build, and endpoint congestion control throttles senders before loss occurs. Correctly tuned, PFC and ECN together keep the fabric lossless without relying on head-of-line-blocking pauses.
RoCE runs on merchant-silicon Ethernet switches from multiple vendors, which broadens the procurement base. For sovereign buyers wary of single-source dependency, that optionality is a strategic property, not a convenience.
Sovereign clusters demand that the interconnect enforce tenancy and residency, not just move bits.
InfiniBand partition keys and RoCE VLAN/VRF isolation carve the physical fabric into logical domains. A model training under one authority cannot address or observe traffic belonging to another sharing the same hardware.
The high-speed training fabric is a local-area interconnect that never leaves the facility. Weights and gradients traverse copper and fiber inside the owner's data center, keeping the most sensitive artifacts physically resident.
The subnet manager and fabric controllers run on infrastructure the owner administers. Routing policy, partitioning, and telemetry are governed in-nation rather than delegated to an external operator.
Fabric topology, routing tables, and partition membership are captured as declarative configuration. The state of the interconnect is inspectable and reproducible, which matters when the cluster is subject to external audit.
At this scale link degradation is expected, so the fabric is instrumented to find and route around faults.
Every port reports error counters, congestion events, and utilization. Silent link degradation, the classic cause of mysterious training slowdowns, is surfaced as a signal rather than diagnosed after a run has already stalled.
The fabric is validated with real collective operations across the full node count, not just point-to-point link tests. Health is defined by the performance of the all-reduce the training job will actually issue.
When a link or switch fails, adaptive and reconfigured routing keep the collective converging over surviving paths. The goal is graceful degradation rather than a hard stop that discards hours of progress.
Firmware, routing policy, and congestion tuning are treated as a controlled, versioned configuration. Changes are staged and reversible, so the interconnect underneath a long training run does not shift unpredictably.