You cannot operate sovereign compute you cannot see. Health, utilization, and interruption are monitored end to end — from the power rail to the training step — under the operator's own eyes.
A hyperscale cluster fails continuously in small ways, so observability starts at the physical layer where those failures originate.
Per-GPU temperature, power draw, memory errors, and clock throttling are collected continuously to catch silicon degrading before it fails a job. A card trending toward failure is flagged for drain rather than discovered by a crashed run.
Link errors, congestion, and port flaps across the training fabric are surfaced, because a single degraded link throttles an entire collective. The network is watched as a first-class component, not assumed healthy.
Power, cooling, and environmental telemetry from the data-center layer are joined with compute health so a thermal event and a throttled node are read as one story. Observability spans the EPC-built facility and the workloads it hosts.
Nodes crossing health thresholds are automatically cordoned and drained so no new work lands on failing hardware. Degradation is contained before it propagates into tenant workloads.
Allocation is not usage — a reserved GPU sitting idle is wasted sovereign capital, and the metrics make that visible.
Reserved accelerator-seconds are compared against measured compute and memory activity, exposing capacity that is held but not worked. Operators see the gap between what is booked and what is doing anything.
Utilization is attributed to tenant, program, and job so a nation can see where its compute is actually going. Consumption is answerable at the granularity of the organizations that own the cluster.
Low occupancy, stalled pipelines, and under-fed accelerators are surfaced as efficiency signals to reclaim wasted capacity. The platform points at waste rather than merely reporting totals.
Power consumption is tied to workloads so the energy cost of sovereign compute is attributable, not aggregate. Each program can be held to account for the energy it draws.
On multi-week training runs, the question is never whether interruptions happen but how fast they are understood and recovered.
Job failures, preemptions, and node losses are detected as they occur and correlated with the underlying hardware or scheduling event. An interruption is tied to its cause in the moment, not reconstructed days later.
The lag between an interruption and a successful checkpoint resume is measured, exposing how much compute a failure actually cost. Recovery is a tracked metric, not an assumption.
Interruptions are classified by root cause — hardware fault, network degradation, preemption, or user error — to distinguish systemic problems from one-off events. Recurring failure modes surface instead of hiding in the noise.
A per-node and per-tenant interruption record accumulates so chronically unreliable hardware and fragile workloads are identified over time. The fleet's reliability is a measured history, not folklore.
For a regulator or an owner, telemetry only matters if it is trustworthy, retained in-nation, and impossible to quietly rewrite.
Metrics, logs, and traces are stored inside the data residency boundary, so operational data never leaves the jurisdiction that owns the cluster. Observability does not become an exfiltration path.
Critical operational events are anchored to a hash-chained log so the record of how the fleet behaved cannot be silently altered. Post-incident review rests on evidence, not on trust in the operator.
Health, utilization, and interruption views are exposed to the owner's own operators rather than mediated by a vendor. The people accountable for the infrastructure see it directly and continuously.
Alerts fire against declared thresholds for saturation, failure rate, and thermal limits so response is triggered by policy, not by someone happening to notice. The conditions that demand action are defined in advance.