X01 Utility voltage sag during a synchronous training load swing
POWCAPCMPSTOTHMRESCG-05CG-06CG-10
Starts with. A transmission-system fault produces a voltage sag at the point of connection that exceeds the UPS input tolerance, followed within seconds by a sustained utility interruption from protection operation upstream.
Service outcomes affected. Training job completion and tolerated loss of progress (performance, recoverability); cluster availability (availability); compute capacity during the generator period (capacity).
Propagation path (9 steps)
- POWSag TRIGGERS UPS transfer from mains or high-efficiency mode to inverter; batteries carry the full IT load including the swing (EXTERNAL_EVENT).
- POWUtility interruption TRIGGERS generator start; UPS battery energy DELAYS consequence for the battery runtime at current state of health (WITHIN the runtime parameter).
- CAP → POWWhen generators close onto the bus and UPS rectifiers walk in, the training load swing CONTRIBUTES_TO repeated large load steps on the generators; synchronous phase alignment across the job AMPLIFIES the step (DEMAND_SURGE).
- POWGenerator governor and AVR response produces frequency and voltage excursions; UPS rectifiers reject the generator source when frequency leaves their input window and return to battery, which CONTRIBUTES_TO further battery depletion (CONTROL_ACTION, AND with step 3).
- POWRepeated rejection AND reduced battery state of health lead to battery exhaustion on one or more UPS blocks before stable generator supply is accepted (SEQUENCE).
- POW → CMPLoss of output on those blocks PROPAGATES_TO the racks they serve; dual-corded racks whose second feed is on a surviving block ride through; single-path rack-scale systems drop.
- CMP → STONode loss in any participant TRIGGERS failure of the whole synchronous job (CASCADE); training progress since the last durable checkpoint is lost.
- THMIf CDU pumps or chilled-water pumps are on the affected blocks or are not UPS-backed, loss of flow PROPAGATES_TO surviving racks as a thermal transient (SHARED_RESOURCE).
- RESRestart of all affected jobs at once creates a restore demand on checkpoint storage (links to X04).
Safeguards, evidence and unknowns
Preconditions
A large synchronous training job is running with periodic power swings between compute and communication or checkpoint phases (swing magnitude and period are capability-profile parameters). UPS modules operate in a high-efficiency mode with a transfer time to inverter. Generator load-acceptance capability was verified with resistive load banks in steps, not with dynamic load. The utility interconnection agreement may contain requirements on load fluctuation.
Operating states
NORMAL → EMERGENCY (on battery, generators starting) → DEGRADED (on generators) → RECOVERY (retransfer to utility) → NORMAL; FAILED for affected blocks if load acceptance fails.
Safeguards that should interrupt it
- Cause side: workload power smoothing (power capping, ramp-rate limiting or on-site energy buffering) that bounds swing magnitude and ramp rate to the plant's characterised acceptance (interrupts step 3).
- Escalation: generator sizing and governor/AVR tuning verified against the measured AI load profile; UPS rectifier walk-in and input-window settings coordinated with generator capability (interrupts step 4).
- Escalation: UPS battery runtime at end-of-life capacity sized for repeated retransfer attempts (DELAYS step 5).
- Consequence side: mechanical-cooling and CDU pumps on UPS-backed supply (interrupts step 8); scheduler-level checkpoint interval matched to tolerated loss (limits step 7).
Evidence that would show the safeguards work
- E3 high-resolution power-quality recordings of the actual training load profile at UPS output and site level (magnitude, ramp, frequency content).
- E4 integrated test of generator acceptance with a dynamic load profile representative of the AI load (programmable load bank or controlled workload), including UPS rectifier behaviour, witnessed (V6).
- E7 generator transient-response study against ISO 8528-5 performance class and the measured profile; UPS coordination study.
- E2 configuration extract of power-capping or ramp-limit settings on the fleet, reconciled to the approved value.
- E3 battery state-of-health measurements and runtime test results.
Typical UNKNOWNs
Generator behaviour under dynamic AI load not tested (U5; US4 on CG-05). Swing profile of the next model generation unknown (U1; US3 unless it changes the envelope, then US4). Utility fault-level and sag statistics not provided (U2; US3).
Where it is assessed
Control families: CAP-04, CAP-05, POW-02, POW-03, POW-05, THM-06, STO-05. Critical gates: CG-05 Resilient electrical supply; CG-06 Thermal continuity; CG-10 Recovery readiness. Required functions: RF-POW-01, RF-POW-02, RF-POW-03, RF-POW-06, RF-CAP-01, RF-THM-02, RF-CMP-01, RF-STO-03.
X02 CDU pump-controller firmware fault causes loss of TCS flow
SUPTHMCMPSTORESCG-06CG-07CG-09CG-14
Starts with. A CDU controller firmware release mishandles a failed differential-pressure sensor reading and commands both duty and standby pumps to a minimum or stopped state, or enters a watchdog reset loop.
Service outcomes affected. Training throughput and completion; compute capacity of affected rows; potential equipment damage if thermal protection is slow or disabled.
Propagation path (7 steps)
- SUP → THMDefective release, deployed through the CDU management tool, is a LATENT condition in all CDUs (COMMON_CAUSE).
- THMSensor fault TRIGGERS incorrect controller action; both pumps stop (M09 then M01 on RF-THM-02); because the standby depends on the same controller, redundancy is DEFEATED.
- THM → CMPLoss of TCS flow PROPAGATES_TO cold plates; low thermal inertia means accelerators reach thermal limits within the TCS ride-through time (a capability-profile parameter, illustratively tens of seconds).
- CMPAccelerators throttle and then hardware thermal protection shuts them down (M23 on RF-CMP-01).
- CMP → STOThe synchronous job fails (CASCADE); progress since the last checkpoint is lost.
- THMFurther CDUs with the same sensor condition fail the same way; failures CORRELATE_WITH the firmware release.
- SUP → RESDiagnosis waits for OEM remote support (DELAYS recovery); manual pump override may be the only immediate recovery.
Safeguards, evidence and unknowns
Preconditions
The same firmware version was deployed to all CDUs of that model across the hall within a short window. Standby pump start depends on the same controller. No hardwired fail-safe keeps a pump running on controller fault. IT-side power capping on coolant-flow loss is not configured. OEM remote support is required to diagnose controller faults.
Operating states
NORMAL (or TRANSITION during rollout) → EMERGENCY → FAILED for affected racks → RECOVERY.
Safeguards that should interrupt it
- Cause side: staged firmware rollout with canary CDUs and defined soak period; pre-deployment test of sensor-fault handling (interrupts step 1).
- Escalation: independent standby pump control or hardwired fail-run on controller fault; pumps default to running at safe speed on loss of control (interrupts step 2).
- Escalation: coolant flow and pressure alarms independent of the CDU controller, feeding the BMS and the IT power-management system to cap power before thermal limits (DELAYS step 3).
- Consequence side: scheduler placement spreading jobs across CDU fault domains where feasible; checkpoint interval (limits step 5).
- Recovery: documented manual override procedure, practised; firmware rollback package held locally.
Evidence that would show the safeguards work
- E4 test record of sensor-fault injection on a non-production CDU with the deployed firmware, showing fail-run behaviour (V5).
- E2 firmware inventory per CDU showing version distribution and rollout waves.
- E4 measured TCS ride-through after pump trip at representative load (time from flow loss to accelerator throttling), V6 under controlled conditions.
- E3 alarm and event history showing flow-loss alarms reach operators and power-capping triggers.
- E1 change record with canary results and approval.
Typical UNKNOWNs
CDU controller failure behaviour on sensor fault not documented by OEM (U2 or U9; US4 on CG-06/CG-07). Whether standby pump control is independent (U10; US4). TCS ride-through at current density (U1; US4).
Where it is assessed
Control families: THM-03, THM-06, SUP-04, SUP-06, OPS-05, CMP-05. Critical gates: CG-06 Thermal continuity; CG-07 Critical control-system integrity; CG-09 Failure containment; CG-14 Change control. Required functions: RF-THM-02, RF-THM-08, RF-THM-01, RF-CMP-01, RF-SUP-01, RF-SUP-03.
X03 PFC storm in a RoCE fabric from a misconfigured switch
NETCMPSTOOPSCG-08CG-09CG-14
Starts with. A switch configuration change maps lossless traffic class priorities inconsistently (for example a DSCP-to-priority mismatch or a buffer threshold error), or a faulty NIC continuously transmits priority flow control (PFC) pause frames.
Service outcomes affected. Collective communication performance; training throughput; job completion across the cluster; checkpoint durability.
Propagation path (6 steps)
- NETMisconfiguration or faulty NIC TRIGGERS persistent pause generation on one port (CONFIG_DRIFT or HUMAN_ACTION).
- NETPause PROPAGATES_VIA inter-switch links hop by hop; head-of-line blocking spreads to unrelated flows sharing the priority; cyclic buffer dependencies can create PFC deadlock (SHARED_RESOURCE, AMPLIFIES).
- NET → CMPCollective operations (all-reduce, all-gather) stall cluster-wide because every rank waits for the slowest (CASCADE).
- CMPCollective timeouts TRIGGER job failures across many jobs, not only the job on the faulty port.
- NET → STOIf storage traffic shares the fabric or priority, checkpoint writes stall (MASKS the last-good checkpoint state).
- OPSTelemetry shows link-level health as normal (links up), which MASKS the cause and DELAYS diagnosis.
Safeguards, evidence and unknowns
Preconditions
The scale-out fabric uses RoCE with PFC for lossless transport. PFC watchdog (storm detection) is disabled or set with a long timer. Congestion control (ECN-based) is tuned for steady state, not for incast during collectives. Configuration templates are pushed by a fabric controller to many switches.
Operating states
NORMAL or TRANSITION (during change) → DEGRADED → FAILED for affected jobs → RECOVERY.
Safeguards that should interrupt it
- Cause side: pre-deployment configuration validation of QoS and PFC consistency across the fabric; staged change with blast-radius limits (interrupts step 1).
- Escalation: PFC watchdog enabled with timers shorter than collective timeouts, disabling PFC or dropping on the offending queue (CONTAINS step 2).
- Escalation: congestion control (ECN marking thresholds, rate control) validated under incast; deadlock-free routing design.
- Consequence side: storage traffic on separate fabric or priority (interrupts step 5).
- Detection: pause-frame counters and queue telemetry with alerts (interrupts step 6).
Evidence that would show the safeguards work
- E2 configuration extract showing PFC watchdog enabled and QoS mapping consistent on all switches, reconciled to the template.
- E4 controlled pause-injection test in a test fabric or isolated partition with the production software version, showing watchdog containment time (V5/V6).
- E3 fabric telemetry history of pause counters and ECN marks during large collectives.
- E1 change records showing staged rollout and validation gates.
Typical UNKNOWNs
Watchdog behaviour on the deployed switch software (U1; US4 on CG-09). Whether storage and compute share a lossless priority (U4 if documents disagree; US3). Deadlock freedom of the routing scheme (U9; US4).
Where it is assessed
Control families: NET-01, NET-02, NET-03, NET-06, OPS-05. Critical gates: CG-08 Compute, storage and network functional integrity; CG-09 Failure containment; CG-14 Change control. Required functions: RF-NET-01, RF-NET-02, RF-NET-03, RF-CMP-03, RF-STO-02.
X04 Checkpoint storage saturation extends recovery beyond tolerance
RESCMPSTONETCAPCG-08CG-10
Starts with. A facility event (for example X01) terminates many training jobs at once; operators and schedulers restart all of them simultaneously.
Service outcomes affected. Recovery time objective for training; tolerated loss of progress; compute capacity utilisation during recovery.
Propagation path (6 steps)
- RES → CMPSimultaneous restart TRIGGERS a read storm against the checkpoint tier (DEMAND_SURGE).
- STORead demand exceeds tier throughput; metadata operations from many ranks opening shards create a metadata hotspot (M15 on RF-STO-03) which AMPLIFIES latency.
- STOBefore the event, retention had filled the tier so recent checkpoint writes failed silently or were truncated; the last verified checkpoint is older than expected (LATENT, CONTRIBUTES_TO progress loss).
- NETStorage fabric congestion from the storm PROPAGATES_TO other tenants and to provisioning traffic (SHARED_RESOURCE).
- STO → CAPRestore time across the cluster exceeds the recovery tolerance; accelerators sit idle (DELAYS recovery).
- CMPTimeouts during restore cause repeated restart attempts that AMPLIFY the storm (positive feedback).
Safeguards, evidence and unknowns
Preconditions
Checkpoint tier sized for steady-state write bandwidth, not for concurrent full-cluster restore. Checkpoint retention policy leaves little free capacity. Metadata service is a single scaling domain. Restart admission is not rate-limited.
Operating states
FAILED → RECOVERY (extended) → DEGRADED → NORMAL.
Safeguards that should interrupt it
- Cause side: restart admission control in the scheduler (staggered, priority-ordered restore) (interrupts step 1).
- Escalation: checkpoint tier and metadata service sized and tested for concurrent restore of the declared job population; tiered checkpointing using node-local storage with asynchronous drain (interrupts step 2).
- Escalation: capacity headroom alarms and retention enforcement; checkpoint completion verification before the previous checkpoint is released (interrupts step 3).
- Consequence side: QoS isolation of storage traffic (CONTAINS step 4); exponential back-off on restart retries (interrupts step 6).
Evidence that would show the safeguards work
- E4 restore test at the declared concurrent scale (or a validated scaled test with T15 extrapolation) measuring time to all jobs running (V6).
- E3 checkpoint write and verification logs showing completion and integrity over a representative period.
- E2 scheduler configuration of restart admission limits.
- E3 capacity forecasting reports with headroom against retention policy.
Typical UNKNOWNs
Restore throughput at full scale (U5; US4 on CG-10). Whether the latest checkpoints are complete (U7; US4). Metadata scaling limit (U1; US3).
Where it is assessed
Control families: STO-02, STO-05, STO-06, CMP-03, RES-04, RES-05. Critical gates: CG-08 Compute, storage and network functional integrity; CG-10 Recovery readiness. Required functions: RF-STO-02, RF-STO-03, RF-STO-01, RF-RES-04, RF-NET-03, RF-CMP-03.
X05 BMS remote-vendor credential compromise alters chilled-water setpoints
SUPSECTHMCMPEVACG-15CG-07CG-06
Starts with. An attacker uses a compromised BMS integrator remote-support credential (shared account, persistent VPN, no MFA) to access the BMS supervisory server.
Service outcomes affected. Training throughput; availability; isolation and security outcomes; evidence and observability outcomes.
Propagation path (7 steps)
- SUP → SECCompromised vendor credential BYPASSES perimeter controls through the vendor conduit (HUMAN_ACTION by adversary).
- SEC → THMAttacker raises chilled-water supply setpoint and widens alarm limits; alarm suppression MASKS the change (M09, M19).
- THMFacility water supply temperature rises gradually; CDU approach temperature consumes margin (DEGRADATION driven by adversary).
- THM → CMPTCS supply temperature rises; accelerators throttle (M07 on RF-CMP-01); throughput degrades without obvious alarm.
- SEC → THMAttacker optionally disables chiller staging or commands CDU valve closure, which ESCALATES to loss of heat rejection.
- CMPThermal shutdown of accelerators; jobs fail; possible equipment stress.
- EVABMS audit logs altered, causing evidence and attribution loss.
Safeguards, evidence and unknowns
Preconditions
Vendor remote access is always on. BMS permits setpoint and alarm-limit changes without second approval. Controllers accept setpoints from the supervisory layer without hard limits. Independent temperature monitoring of TCS and facility water does not exist outside the BMS.
Operating states
NORMAL → DEGRADED (slow-burn) → EMERGENCY → RECOVERY; analysis shall also consider initiation during MAINTENANCE.
Safeguards that should interrupt it
- Cause side: just-in-time vendor access with MFA, approval per session and session recording (interrupts step 1).
- Escalation: setpoint hard limits enforced in field controllers, not only in the supervisory layer; two-person approval for setpoint and alarm-limit changes (interrupts step 2).
- Detection: independent temperature and flow telemetry from IT-side sensors (accelerator inlet coolant temperature) correlated with BMS values; configuration change detection on BMS (interrupts steps 2–4).
- Consequence side: IT power capping on coolant temperature (DELAYS step 6).
- Evidence: write-once off-box logging of BMS changes (interrupts step 7).
Evidence that would show the safeguards work
- E2 remote-access configuration showing JIT activation, MFA and session recording; E3 sample of session records reconciled to work orders.
- E4 test that a setpoint outside hard limits is rejected by the field controller (V5, non-production or authorised window).
- E3 correlation between IT-side coolant telemetry and BMS readings over a representative period.
- E5 independent OT security assessment against IEC 62443-3-2 zones and conduits.
Typical UNKNOWNs
Whether vendor accounts are unique and revoked on staff change (U2; US4 on CG-15). Whether controllers enforce limits (U1; US4 on CG-07). Integrity of BMS logs (U7; US2–US3).
Where it is assessed
Control families: SUP-06, SEC-01, SEC-02, SEC-03, SEC-05, THM-06, EVA-02. Critical gates: CG-15 Integrated digital authority over infrastructure; CG-07 Critical control-system integrity; CG-06 Thermal continuity. Required functions: RF-SEC-01, RF-SEC-02, RF-SEC-05, RF-THM-08, RF-THM-04, RF-THM-03, RF-CMP-01, RF-EVA-01.
X06 Lithium-ion BESS or UPS battery thermal runaway
POWPHYCMPENVSUPRESCG-04CG-05CG-10
Starts with. An internal short circuit in a lithium-ion cell of a UPS battery cabinet or BESS module initiates thermal runaway.
Service outcomes affected. Safety outcomes (primary); availability and capacity of the affected block; recovery outcomes; environmental compliance.
Propagation path (8 steps)
- POWCell runaway releases flammable and toxic off-gas before visible fire (M25 on RF-POW-03).
- POW → PHYOff-gas accumulates; without detection and ventilation the room atmosphere can reach deflagration conditions (ESCALATION; CG-04).
- POWHeat PROPAGATES_TO adjacent cells and modules unless module design and spacing CONTAIN it.
- PHYFire detection and suppression act; suppression may not stop cell-level runaway (re-ignition risk); smoke PROPAGATES_VIA shared ventilation to adjacent spaces.
- PHY → POWEmergency response requires de-energising the UPS and its battery; the UPS block loses ride-through, so the load is either transferred to bypass (exposed to utility disturbances) or shut down (CONTROL_ACTION or HUMAN_ACTION).
- POW → CMPRacks on that block operate without ride-through or are lost; synchronous jobs fail if any participant is affected.
- ENVElectrolyte, combustion products and suppressant residue contaminate equipment; environmental release reporting.
- SUP → RESReplacement battery modules and cabinet repair have long lead times (DELAYS recovery to NORMAL).
Safeguards, evidence and unknowns
Preconditions
Battery installed in a room adjacent to electrical rooms or data hall. Off-gas detection may not be fitted; battery management system (BMS-battery) alarms are routed to the UPS, not to the fire system. Fire-test evidence for propagation (for example UL 9540A test reports) may not match the installed configuration. Emergency response plan does not address re-ignition.
Operating states
NORMAL → EMERGENCY → SHUTDOWN (battery string and possibly UPS) → DEGRADED (UPS without battery) → RECOVERY (long, with replacement).
Safeguards that should interrupt it
- Cause side: cell and module condition monitoring with thermal and voltage anomaly detection; manufacturer quality and approved sourcing (interrupts step 1).
- Escalation: off-gas detection triggering ventilation and alarm; explosion control (venting or exhaust) per the installation's hazard analysis (interrupts step 2).
- Escalation: module-level propagation resistance evidenced by large-scale fire testing of the installed configuration; separation distances and fire-rated enclosure (CONTAINS step 3).
- Consequence side: fire strategy and emergency response plan specific to lithium-ion, including fire-service liaison, re-ignition watch and safe de-energisation (CONTAINS steps 4–5).
- Consequence side: electrical topology where loss of one battery string does not remove ride-through from the whole load (limits step 6).
Evidence that would show the safeguards work
- E5 large-scale fire test report for the installed battery product and configuration, reviewed by a competent fire engineer against the installation.
- E7 hazard mitigation analysis and explosion-control design calculation for the room.
- E4 functional test of off-gas detection and ventilation interlocks (V5).
- E1/E4 emergency response plan and exercise record with the fire service (V4).
- E3 battery management system telemetry and alarm routing configuration (E2).
Typical UNKNOWNs
Applicability of fire-test results to the installed configuration (U9; US4 on CG-04). Whether off-gas alarms reach the fire system (U10; US4). Replacement lead time (U2; US3 on CG-10).
Where it is assessed
Control families: POW-03, PHY-06, ENV-05, OPS-02, RES-06, SUP-05. Critical gates: CG-04 Life safety and hazardous energy; CG-05 Resilient electrical supply; CG-10 Recovery readiness. Required functions: RF-POW-03, RF-PHY-04, RF-ENV-04, RF-OPS-01, RF-RES-03, RF-SUP-02.
X07 Coolant leak at a rack manifold or quick-disconnect
THMPOWCMPPHYOPSRESCG-04CG-06CG-09
Starts with. A quick-disconnect seal fails or a rack manifold cracks, often during a pressure transient (pump start, valve closure) or after a maintenance reconnection.
Service outcomes affected. Safety (electrical hazard); availability and capacity of affected racks; physical damage; recovery.
Propagation path (7 steps)
- THMPressure transient TRIGGERS seal failure at a degraded quick-disconnect (DEGRADATION, CONTRIBUTES_TO); coolant escapes (M25 on RF-THM-06).
- THMAutomatic make-up maintains loop pressure, which MASKS the leak at the CDU and sustains the leak volume.
- THM → POWCoolant reaches power shelves or busbar connections; ESCALATION to electrical fault, arc or tracking (CG-04).
- POWRack power protection trips; if protection is slow or the fault involves the busway, upstream protection clears a larger section (M05 if miscoordinated).
- POW → CMPRack and possibly row loss; synchronous jobs fail.
- PHYCoolant spreads under or between racks to adjacent equipment; floor-level detection alarms only after pooling (DELAYS detection).
- OPS → RESIsolation, drying, inspection and requalification of affected equipment before re-energisation (RECOVERY); risk that re-energising wet equipment REINTRODUCES the fault.
Safeguards, evidence and unknowns
Preconditions
Rack manifold sits above power shelves or busbar connections. Leak detection is a single sensing cable at floor level. The CDU automatically makes up coolant from a reservoir when pressure falls. Per-rack isolation valves are manual.
Operating states
NORMAL or MAINTENANCE → EMERGENCY → DEGRADED (rack isolated) → RECOVERY.
Safeguards that should interrupt it
- Cause side: quick-disconnect inspection and replacement interval; pressure-test after any reconnection; soft-start and controlled valve timing to limit transients (interrupts step 1).
- Detection: point leak sensors at manifold connections and drip trays; CDU make-up volume and rate alarm and limit (interrupts step 2).
- Escalation: physical separation or shielding of fluid connections from energised conductors; automatic per-rack isolation on leak detection (interrupts step 3).
- Consequence side: selective protection coordination at rack and busway level (CONTAINS step 4).
- Recovery: re-energisation procedure with insulation testing and competent sign-off (interrupts step 7).
Evidence that would show the safeguards work
- E4 leak-detection functional test with a defined fluid, volume and location at manifold level, recording time to alarm and isolation (V5).
- E2 CDU configuration showing make-up limit and alarm settings.
- E4 commissioning pressure-test records per rack and after maintenance reconnections.
- E7 protection coordination study covering rack power shelves and busway taps.
- E1 re-energisation procedure; E6 interview evidence that staff know it.
Typical UNKNOWNs
Behaviour of make-up logic during leak (U1; US4 on CG-06). Leak detection coverage at manifold height (U5; US4 on CG-04). QD condition across fleet (U5; US3).
Where it is assessed
Control families: THM-03, THM-05, POW-04, OPS-02, PHY-03. Critical gates: CG-04 Life safety and hazardous energy; CG-06 Thermal continuity; CG-09 Failure containment. Required functions: RF-THM-06, RF-THM-02, RF-POW-04, RF-POW-05, RF-CMP-01, RF-OPS-02.
X08 Facility water contamination fouls heat exchangers and cold plates
ENVTHMCMPPHYOPSCG-06CG-04
Starts with. A water-treatment failure in an open cooling-tower circuit (dosing pump fault, missed chemistry checks) or construction debris from an expansion tie-in enters the facility water loop.
Service outcomes affected. Training throughput at peak load; resilient cooling capacity; safety; equipment life.
Propagation path (6 steps)
- ENV → THMChemistry excursion or debris CAUSES fouling and biological growth in the facility loop (DEGRADATION; M07 on RF-THM-07).
- THMFouling of CDU heat-exchanger plates increases approach temperature (M07 on RF-THM-03) across many CDUs fed by the same loop (COMMON_CAUSE).
- THM → CMPTCS supply temperature rises at a given load; accelerators throttle at peak load (M07 on RF-CMP-01).
- THMIf TCS is contaminated (HX breach, improper fill or material incompatibility), cold-plate microchannels foul or corrode, which ESCALATES to localised overheating and possible leaks.
- PHY → ENVBiological growth in cooling towers creates a water-borne pathogen hazard (ESCALATION to CG-04).
- OPSCleaning requires taking CDUs out of service, which reduces resilient capacity (MAINTENANCE state).
Safeguards, evidence and unknowns
Preconditions
Facility water passes through CDU heat exchangers with tight approach-temperature design. Side-stream filtration is undersized or bypassed. Chemistry monitoring is manual and periodic. Flushing and cleanliness requirements for tie-ins are not enforced. TCS fill water and inhibitor are managed by a different team.
Operating states
NORMAL → DEGRADED (slow-burn) → EMERGENCY if approach collapses → MAINTENANCE (cleaning) → RECOVERY.
Safeguards that should interrupt it
- Cause side: water-treatment programme with continuous chemistry monitoring and alarms; filtration and strainers; flushing and cleanliness acceptance for tie-ins (interrupts step 1).
- Detection: approach-temperature trending per CDU heat exchanger with alarm on drift (interrupts step 2).
- Escalation: TCS chemistry and material compatibility management with periodic sampling (interrupts step 4).
- Escalation: cooling-tower pathogen risk management per applicable regulation (interrupts step 5).
- Consequence side: N+1 CDU capacity allowing cleaning without loss of outcome (limits step 6).
Evidence that would show the safeguards work
- E3 chemistry monitoring records and laboratory results over a representative period.
- E3 approach-temperature trend data per CDU with alarm configuration (E2).
- E4 tie-in flushing and cleanliness acceptance records.
- E5 independent water-treatment audit.
Typical UNKNOWNs
TCS chemistry history (U1; US3). Material compatibility of mixed-vendor cold plates and manifolds (U9; US3). Fouling rate (U8; US2).
Where it is assessed
Control families: THM-05, THM-04, ENV-03, OPS-03, THM-01. Critical gates: CG-06 Thermal continuity; CG-04 Life safety and hazardous energy. Required functions: RF-THM-07, RF-THM-03, RF-THM-04, RF-ENV-02, RF-CMP-01, RF-OPS-04.
X09 Shared fibre route cut defeats diverse connectivity
PHYDEPNETCAPSTOSECSUPEVACG-03CG-09CG-10
Starts with. Third-party excavation near the site cuts a duct carrying fibre for two carriers that were contracted as diverse.
Service outcomes affected. Inference availability and latency; recovery-point outcome for training; observability.
Propagation path (7 steps)
- PHY → DEPExternal excavation TRIGGERS a cut of the shared duct (EXTERNAL_EVENT; COMMON_CAUSE through the shared route).
- NETBoth carriers' circuits fail; diversity DEFEATED (M01 on RF-NET-03 frontend and WAN).
- NET → CAPInference ingress lost; inference Service Outcome breached immediately.
- NET → STOInter-site checkpoint replication stops; recovery-point exposure grows (DELAYS replication).
- NET → SECCloud-hosted identity or MFA unreachable; operators cannot authenticate to some systems (links to X15).
- NET → SUPRemote vendor support unavailable during any concurrent incident (SHARED_RESOURCE).
- EVARemote monitoring of the site lost; site-local evidence must be retained.
Safeguards, evidence and unknowns
Preconditions
Diversity was accepted on carrier assurance without route evidence. Both carriers' circuits enter through different building entries but share a duct or bridge crossing outside the boundary. Inference ingress, inter-site checkpoint replication, remote vendor access and cloud identity services all depend on these circuits. Out-of-band access has no independent path.
Operating states
NORMAL → DEGRADED or FAILED (external connectivity) → RECOVERY (carrier repair).
Safeguards that should interrupt it
- Cause side: route-level diversity verified with carrier route evidence to a defined separation, including outside-plant segments (interrupts step 2).
- Escalation: third diverse path using different medium (for example a different route or wireless backup) for critical control, OOB and identity traffic (CONTAINS steps 5–6).
- Consequence side: inference failover to another site with capacity reserved (limits step 3); local identity fallback (limits step 5).
Evidence that would show the safeguards work
- E5 carrier route documentation with separation evidence, reviewed against physical survey.
- E4 failover test of inference ingress to an alternate site (V6).
- E4 test that OOB and identity fallback work with all carrier circuits disabled (V5/V6, in an authorised window).
Typical UNKNOWNs
Carrier route geometry (U2; US4 on CG-03 when diversity is credited). Dependence of identity on WAN (U10; US5 if it affects CG-10 and CG-15).
Where it is assessed
Control families: DEP-02, DEP-03, NET-05, NET-04, RES-03. Critical gates: CG-03 Critical dependency visibility; CG-09 Failure containment; CG-10 Recovery readiness. Required functions: RF-NET-03, RF-NET-04, RF-DEP-02, RF-STO-05, RF-SEC-01, RF-SUP-03.
X10 Firmware supply-chain defect across the GPU fleet
SUPCMPSTOCAPRESCG-08CG-14CG-10
Starts with. An authentic, correctly signed accelerator firmware or driver release contains a defect that, under a specific power or clock state, produces silent data corruption or uncorrectable memory errors.
Service outcomes affected. Data and model integrity; training completion; compute capacity during rollback.
Propagation path (6 steps)
- SUP → CMPDefective release is deployed to the fleet; signature validation passes, so authenticity checks do not interrupt it (LATENT, COMMON_CAUSE).
- CMPUnder the triggering power state, some accelerators compute incorrect results without error indication (M16 / M09 on RF-CMP-01).
- CMP → STOCorrupted gradients are incorporated into model state; checkpoints written after onset are corrupted (PROPAGATES_TO RF-STO-04).
- CAPTraining loss anomalies appear later; detection lag means older good checkpoints may already have been rotated out (DELAYS detection; ESCALATION of progress loss).
- CMPUncorrectable errors on other devices cause node failures and job restarts that CORRELATE_WITH the release.
- RESRecovery requires rollback of firmware across the fleet and re-training from the last verified checkpoint; rollback package availability and compatibility determine duration.
Safeguards, evidence and unknowns
Preconditions
Release passes signature validation and functional smoke tests. Rollout policy allows fleet-wide deployment within a short window. Silent data corruption screening is not part of acceptance. Checkpoint retention keeps only a few recent checkpoints. Fleet has no firmware diversity.
Operating states
TRANSITION (rollout) → NORMAL (latent) → DEGRADED (detected) → RECOVERY (rollback and re-training from a known-good checkpoint).
Safeguards that should interrupt it
- Cause side: staged rollout with canary population and soak period including SDC screening tests and burn-in under representative power states (interrupts step 1).
- Detection: periodic SDC detection tests and cross-replica result comparison; training-health monitoring (loss and gradient anomalies) with alerts (interrupts steps 2–4).
- Consequence side: retention of verified checkpoints spanning more than the expected detection lag; checkpoint integrity validation (limits step 4).
- Recovery: tested rollback package and procedure (interrupts step 6).
Evidence that would show the safeguards work
- E1/E4 rollout records showing canary waves, soak duration and SDC test results.
- E2 fleet firmware inventory by version and wave.
- E4 rollback test on a representative partition (V5).
- E3 training-health monitoring alerts and responses.
Typical UNKNOWNs
OEM defect disclosure (U2/U6; US3). SDC test coverage of triggering conditions (U8; US4 on CG-08). Detection lag (U1; US3).
Where it is assessed
Control families: SUP-04, CMP-04, CMP-05, STO-03, STO-05, OPS-05. Critical gates: CG-08 Compute, storage and network functional integrity; CG-14 Change control; CG-10 Recovery readiness. Required functions: RF-SUP-01, RF-CMP-04, RF-CMP-01, RF-CMP-05, RF-STO-04, RF-STO-05, RF-RES-04.
X11 Concurrent maintenance exposes a hidden shared electrical dependency
OPSPOWCMPTHMRESCG-02CG-05CG-14
Starts with. The A-side UPS and distribution path is taken out of service for planned maintenance.
Service outcomes affected. Availability and capacity during maintenance (concurrent-maintainability claim); training completion.
Propagation path (6 steps)
- OPSPlanned isolation of the A path (HUMAN_ACTION, ENABLING).
- POWRacks whose both cords are on A lose power (M29 misallocation, LATENT; CAUSES M01 on RF-POW-05 for those racks).
- POWB path carries all remaining load; at peak training load B-side PDUs or busway taps exceed rating and trip (M14), which CASCADES to further rack loss.
- POW → CMPSynchronous jobs on affected racks fail; job loss extends to all jobs with any participant on those racks.
- CMP → THMSudden load drop and then restart surge cause cooling-plant control transients (AMPLIFIES).
- OPS → RESMaintenance must be aborted and A path restored under time pressure, raising the risk of switching error (links to X16).
Safeguards, evidence and unknowns
Preconditions
Racks are designed dual-corded A/B, but an as-built divergence exists: some rack power shelves have both inputs connected to A-side PDUs, or rack-scale systems with single-input power shelves were added on the A side. The B path's capacity was checked against design load, not measured peak AI load. The method statement does not require verification of cord-to-feed mapping or B-side headroom immediately before isolation.
Operating states
NORMAL → MAINTENANCE → FAILED (affected racks) → RECOVERY.
Safeguards that should interrupt it
- Cause side: periodic as-built audit of cord-to-feed mapping reconciled in EPMS, and per-outlet metering that detects single-path loading (interrupts step 2).
- Cause side: pre-isolation readiness check in the method statement: measured B-side load and headroom at expected peak, list of single-path racks, workload drain or power capping applied (interrupts steps 2–3).
- Escalation: power capping on B-side overload risk (interrupts step 3).
- Consequence side: scheduler drain of single-path racks before maintenance (limits step 4).
Evidence that would show the safeguards work
- E2 EPMS per-outlet load and feed mapping export reconciled to the rack power register (RECONCILIATION).
- E1 method statement with readiness checks; E3 record of the checks performed for the last maintenance events.
- E4 concurrent-maintenance test with representative load (V6).
Typical UNKNOWNs
Cord-to-feed mapping for racks added since commissioning (U3; US5 where it bears on both CG-02 and CG-05, otherwise US4 on CG-05). Measured peak B-side load (U1; US4).
Where it is assessed
Control families: OPS-02, OPS-05, POW-04, POW-06, DEP-01, EVA-02. Critical gates: CG-02 Boundary and architecture integrity; CG-05 Resilient electrical supply; CG-14 Change control. Required functions: RF-POW-05, RF-POW-04, RF-DEP-01, RF-OPS-02, RF-OPS-03, RF-EVA-02, RF-CMP-01.
X12 Heatwave combined with water restriction on evaporative heat rejection
ENVDEPTHMCMPPOWRESCG-06CG-03CG-01
Starts with. A prolonged heatwave raises ambient wet-bulb temperature above the heat-rejection design condition while the water utility imposes abstraction or supply restrictions.
Service outcomes affected. Compute capacity and throughput; availability of lower-priority workloads; environmental compliance (water).
Propagation path (6 steps)
- ENVAmbient exceedance (EXTERNAL_EVENT) CAUSES reduced heat-rejection capacity (M07 on RF-THM-04).
- ENV → DEPWater restriction CONTRIBUTES_TO further capacity loss as evaporative duty cannot be sustained (AND with step 1; M14 on RF-ENV-02).
- THMChillers operate at high condensing pressure; some trip on high-pressure protection (M24 if spurious, M01 if genuine), consuming redundancy.
- THM → CMPFacility water and TCS supply temperatures rise; accelerators throttle; at the limit, load must be shed.
- POW → DEPGrid demand-response obligations and higher cooling power CONFLICT_WITH sustaining full IT load.
- RESLoad shedding by priority (graceful degradation) must be executed; if not pre-planned, ad hoc decisions DELAY response.
Safeguards, evidence and unknowns
Preconditions
Evaporative cooling towers or adiabatic coolers provide most heat rejection. Design ambient was based on historical data without climate projection. On-site water storage autonomy is limited. The grid operator issues demand-response requests in the same period.
Operating states
NORMAL → DEGRADED (derated) → EMERGENCY (load shedding) → RECOVERY.
Safeguards that should interrupt it
- Cause side: heat-rejection design and derating at projected extreme ambient; hybrid dry/wet operation (interrupts step 1).
- Escalation: water storage autonomy sized for the restriction scenario; alternative supply agreement (interrupts step 2).
- Consequence side: pre-approved load-shedding priorities and automated power capping by workload class (limits steps 4–6).
Evidence that would show the safeguards work
- E7 heat-rejection capacity analysis at projected extreme ambient, with derating curves from equipment data.
- E3 performance during the hottest observed periods (V2).
- E1 water-supply agreement and storage autonomy calculation.
- E4 exercise of load-shedding priorities (V4/V6).
Typical UNKNOWNs
Extreme-ambient design basis under climate projection (U9; US3–US4). Utility restriction policy (U2; US3).
Where it is assessed
Control families: ENV-06, ENV-03, THM-01, THM-04, RES-01, CAP-05. Critical gates: CG-06 Thermal continuity; CG-03 Critical dependency visibility; CG-01 Capability and outcome definition. Required functions: RF-THM-04, RF-ENV-02, RF-THM-03, RF-CMP-01, RF-RES-02, RF-CAP-02.
X13 Time-source failure causes divergence across control planes
NETSECSTOCMPEVACG-08CG-11CG-15
Starts with. GNSS signal loss or interference at the grandmaster antennas, or a misconfigured clock wins best-master selection and steps time.
Service outcomes affected. Availability; data integrity; observability and evidence.
Propagation path (5 steps)
- NETGNSS loss TRIGGERS holdover; drift accumulates; or a rogue clock is selected (M09 on RF-NET-05).
- NET → SECClock offset beyond tolerance causes certificate and token validation failures; operators and services fail to authenticate (PROPAGATES_TO RF-SEC-01).
- NET → STODistributed storage leases and consensus behave incorrectly; possible metadata inconsistency (M27).
- NET → CMPScheduler timeouts and job failures from skewed heartbeats.
- NET → EVALogs and EPMS sequence-of-events records lose correlation, causing evidence and attribution loss for any concurrent event (MASKS causal reconstruction).
Safeguards, evidence and unknowns
Preconditions
PTP and NTP hierarchy depends on one GNSS source type; holdover oscillators are limited; clients permit large time steps; the same time hierarchy serves the AI fabric, storage, identity, BMS/EPMS and logging.
Operating states
NORMAL → DEGRADED (holdover or wrong time) → RECOVERY.
Safeguards that should interrupt it
- Cause side: multiple diverse time sources with authenticated distribution and grandmaster priority settings restricting which clocks can become master (interrupts step 1).
- Escalation: holdover oscillator quality sized for the expected outage; client policy to slew rather than step beyond threshold; offset monitoring at every tier with alarms (interrupts steps 2–4).
- Evidence: logs carry source and synchronisation status; EPMS has local high-accuracy time-stamping (limits step 5).
Evidence that would show the safeguards work
- E2 time-hierarchy configuration including master priorities and client step policy.
- E4 GNSS-loss test in holdover with measured drift over the declared holdover period (V5).
- E3 offset monitoring history.
Typical UNKNOWNs
Which systems depend on which time source (U10; US4–US5 depending on gates affected). Holdover performance (U1; US3).
Where it is assessed
Control families: NET-06, SEC-01, STO-03, EVA-02. Critical gates: CG-08 Compute, storage and network functional integrity; CG-11 Evidence sufficiency; CG-15 Integrated digital authority over infrastructure. Required functions: RF-NET-05, RF-SEC-01, RF-STO-04, RF-CMP-03, RF-POW-07, RF-EVA-01.
X14 Recovery reintroduces a corrupt configuration or image
RESNETCMPTHMCG-10CG-14CG-07
Starts with. After an incident caused by a defective configuration change (for example X03) or a defective image, operators restore from the most recent backup.
Service outcomes affected. Recovery time; availability; integrity of configuration.
Propagation path (4 steps)
- RESRecovery selects the latest backup (HUMAN_ACTION under time pressure).
- RES → NET → CMP → THMRestore REINTRODUCES the defective configuration or image (RECOVERY_INDUCED).
- NET → CMPOriginal failure recurs; or an older BMS backup removes interlocks added after an earlier incident (M16 on RF-SEC-05 or RF-THM-08).
- RESRepeated failure extends recovery time beyond tolerance and erodes confidence in recovery (DELAYS).
Safeguards, evidence and unknowns
Preconditions
Backups are taken on schedule after every change, so the latest backup contains the defect. There is no "known good" tag. The golden image repository has no validation gate. BMS controller backups contain outdated logic or setpoints.
Operating states
FAILED → RECOVERY → FAILED (again) → RECOVERY.
Safeguards that should interrupt it
- Cause side: versioned configuration with known-good tagging and change correlation; restore procedure selects the last validated baseline, not the latest backup (interrupts step 1).
- Escalation: validation of restored configuration or image in a staging environment or canary before fleet application (interrupts step 2).
- Escalation: BMS and controller backups version-controlled and reconciled to current approved logic (interrupts step 3).
Evidence that would show the safeguards work
- E2 configuration repository showing tagged known-good baselines and correlation to changes.
- E4 restore exercise selecting a known-good baseline and validating it (V5/V6).
- E1 recovery procedure with validation steps.
Typical UNKNOWNs
Whether controller backups match the current approved logic (U3; US4 on CG-07). Whether known-good baselines exist for all device classes (U5; US3).
Where it is assessed
Control families: RES-04, OPS-05, CMP-06, NET-06, THM-06. Critical gates: CG-10 Recovery readiness; CG-14 Change control; CG-07 Critical control-system integrity. Required functions: RF-RES-04, RF-RES-03, RF-OPS-03, RF-CMP-06, RF-NET-02, RF-THM-08.
X15 Shared identity service failure blocks operators and break-glass access during an incident
SECOPSTHMPOWCMPRESCG-15CG-10CG-13
Starts with. The central identity provider or MFA service becomes unavailable (for example through a power event in the hall hosting it, a WAN loss as in X09, or a certificate expiry).
Service outcomes affected. Recovery time of any concurrent incident; availability; control-plane integrity.
Propagation path (5 steps)
- SECIdentity service failure (M01 on RF-SEC-01), possibly CASCADING from a power or network event.
- SEC → OPSOperators cannot log in to BMS, EPMS or cluster managers (SHARED_RESOURCE).
- SECBreak-glass procedure depends on the same service or fault domain; break-glass is DEFEATED.
- OPS → THM → POW → CMPResponse to any concurrent incident is DELAYED; automated systems continue unsupervised.
- RESRecovery of the identity service may itself require privileged access that depends on the identity service (circular recovery dependency).
Safeguards, evidence and unknowns
Preconditions
BMS, EPMS, cluster managers, jump hosts and fabric controllers authenticate against the same identity service. Break-glass accounts are stored in a vault that itself depends on the identity service or is hosted in the same fault domain. Local console access to OT HMIs has been removed.
Operating states
Any → EMERGENCY (if concurrent incident) → RECOVERY.
Safeguards that should interrupt it
- Cause side: identity service deployed across independent fault domains, with local authentication caches at critical systems (interrupts step 2).
- Escalation: break-glass credentials held offline in physical custody, independent of the identity service, tested on a schedule (interrupts step 3).
- Escalation: local operator panels for safety-relevant OT functions (interrupts step 4).
- Recovery: documented, non-circular identity recovery procedure (interrupts step 5).
Evidence that would show the safeguards work
- E4 break-glass test with the identity service disabled (V5, authorised window).
- E2 identity-service deployment map against fault domains.
- E1 identity recovery procedure; E4 exercise record (V4).
Typical UNKNOWNs
Dependence of break-glass on identity (U10; US4 on CG-15, US5 if CG-10 also affected). Hosting fault domain of identity components (U10; US4).
Where it is assessed
Control families: SEC-01, NET-04, RES-04, DEP-02, OPS-01. Critical gates: CG-15 Integrated digital authority over infrastructure; CG-10 Recovery readiness; CG-13 Responsibility and authority. Required functions: RF-SEC-01, RF-OPS-01, RF-RES-03, RF-NET-04, RF-GOV-01.
X16 Alarm flood during an electrical event leads to operator mis-switching
POWOPSTHMCMPRESCG-07CG-13CG-05
Starts with. A medium-voltage fault causes loss of one utility feed and multiple automatic transfers, generating thousands of BMS and EPMS alarms within minutes.
Service outcomes affected. Availability; safety (switching error near energised equipment); recovery time.
Propagation path (5 steps)
- POWFault TRIGGERS protection and transfers; alarm flood follows (M20 on RF-OPS-01; DEMAND_SURGE on operators).
- OPSAlarm flood MASKS the root alarm; operator misidentifies which side is healthy (HUMAN_ACTION; systemic contributors: interface, alarm load, staffing).
- OPS → POWOperator opens the wrong breaker during a manual transfer, dropping the remaining healthy side (M09 on RF-OPS-02).
- POW → THM → CMPLoad lost across the affected block; cooling pumps and CDUs on that block stop; synchronous jobs fail.
- RESRecovery sequence executed under stress; risk of further error.
Safeguards, evidence and unknowns
Preconditions
Alarm rationalisation has not been performed; there is no first-out or state-based suppression. Emergency operating procedures are generic. The single-line diagram on the HMI does not match the as-built after recent changes. Staffing at night is minimal.
Operating states
EMERGENCY → FAILED (if mis-switching) → RECOVERY.
Safeguards that should interrupt it
- Cause side: alarm rationalisation and state-based suppression with first-out indication (interrupts step 2).
- Escalation: event-specific emergency operating procedures with decision aids; two-person verification for switching; key interlocks preventing unsafe breaker combinations (interrupts step 3).
- Escalation: HMI single-line diagram reconciled to as-built after every change (interrupts step 2).
- Competence: simulator or tabletop training for rare transfer scenarios.
Evidence that would show the safeguards work
- E3 alarm-rate statistics during past events against the rationalisation target.
- E1 emergency operating procedures; E4 exercise records (V4).
- E1/E2 interlock schemes and HMI drawing revision control reconciled to as-built.
Typical UNKNOWNs
HMI as-built accuracy (U3; US3–US4). Night-shift competence for manual transfer (U5; US3).
Where it is assessed
Control families: OPS-04, OPS-01, OPS-02, POW-06, EVA-02. Critical gates: CG-07 Critical control-system integrity; CG-13 Responsibility and authority; CG-05 Resilient electrical supply. Required functions: RF-OPS-01, RF-OPS-02, RF-OPS-05, RF-POW-07, RF-EVA-02.
X17 Fire in a shared compartment disables paired A/B routes and triggers cascading shutdowns
PHYPOWNETTHMCMPSTORESCG-04CG-09CG-05
Starts with. A cable fire in a riser or an electrical room in which both A and B feeders (or both fabric trunk routes) pass.
Service outcomes affected. Safety; availability; capacity; recovery time; physical damage.
Propagation path (6 steps)
- PHYCable fault ignites cables (M25); smoke PROPAGATES_VIA the shared compartment.
- PHY → POW → NETBoth A and B routes are damaged (COMMON_CAUSE through shared compartment; redundancy DEFEATED).
- PHY → THMFire-alarm interface shuts down air handlers in the adjacent hall (CONTROL_ACTION; CONFLICTS_WITH continuity cooling).
- PHY → POWEPO initiated over a wider area than the fire zone (M24 spurious extent; M05).
- CMP → STOLarge-scale loss of compute and storage; unclean shutdown of storage systems.
- RESRe-entry, damage assessment and cable replacement extend recovery; contamination cleaning of equipment.
Safeguards, evidence and unknowns
Preconditions
Route segregation is by tray, not by fire compartment. The fire-alarm cause-and-effect matrix shuts down air handlers and, in one zone, initiates EPO across a wider area than intended. Clean-agent discharge in an adjacent hall requires evacuation and delays re-entry.
Operating states
NORMAL → EMERGENCY → FAILED → RECOVERY (extended by damage).
Safeguards that should interrupt it
- Cause side: segregation of A/B routes into separate fire compartments or fire-rated enclosures (interrupts step 2).
- Detection: early-warning smoke detection suited to high-airflow spaces (DELAYS escalation).
- Escalation: cause-and-effect matrix verified by integrated test so that shutdown actions are limited to the affected zone (interrupts steps 3–4).
- Consequence side: storage systems with protected write caches and orderly-shutdown signals (limits step 5).
Evidence that would show the safeguards work
- E2/E1 route survey showing compartment of each A/B route, reconciled to drawings.
- E4 integrated fire-alarm cause-and-effect test record (V6) witnessed by fire and electrical engineers.
- E5 fire-strategy review by a competent fire engineer.
Typical UNKNOWNs
Whether A/B routes share a compartment at any point (U10; US5 if both CG-05 and CG-09 are affected). Actual extent of EPO zones (U4 if drawings and logic disagree; US4 on CG-04).
Where it is assessed
Control families: PHY-03, PHY-06, POW-04, NET-05, RES-02. Critical gates: CG-04 Life safety and hazardous energy; CG-09 Failure containment; CG-05 Resilient electrical supply. Required functions: RF-PHY-04, RF-POW-05, RF-NET-06, RF-THM-05, RF-RES-01, RF-RES-03.
X18 Automated remediation or AI-enabled agent performs a mass action
EVACMPSECPOWTHMSTOCG-15CG-09CG-05
Starts with. A telemetry pipeline defect generates false accelerator error events across a large fraction of nodes; an automated remediation system (or AI-enabled operations agent) drains and reboots the affected nodes, or applies a fleet-wide power cap, in one action.
Service outcomes affected. Availability; training completion; recovery time; electrical and thermal stability.
Propagation path (6 steps)
- EVA → CMPTelemetry defect produces false indications (M18 on RF-CMP-05).
- SECAutomation acts on the false indications at fleet scale (CONTROL_ACTION; M08 excess output on RF-SEC-03).
- SEC → CMPMass drain and reboot terminates many jobs (CASCADE).
- CMP → POWSudden load drop followed by restart surge creates large load steps at UPS and, if on generator, at the generators (DEMAND_SURGE; links to X01).
- CMP → THMCooling-plant control responds to rapid load change; overshoot or undershoot of supply temperature (M13 on RF-THM-08).
- CMP → STORestart storm on provisioning and checkpoint storage (links to X04).
Safeguards, evidence and unknowns
Preconditions
Automation has authority to act on any number of nodes; no blast-radius limit or rate limit; no human approval above a threshold; telemetry inputs are not validated for plausibility; power-cap changes have no ramp-rate limit.
Operating states
NORMAL → DEGRADED → FAILED (jobs) → RECOVERY.
Safeguards that should interrupt it
- Cause side: plausibility checks on telemetry before automated action (interrupts step 1).
- Escalation: blast-radius and rate limits on automated actions; human approval above a node-count or power threshold; command allowlisting (interrupts step 2).
- Escalation: ramp-rate limits on fleet power-cap changes (interrupts step 4).
- Consequence side: restart admission control (limits step 6).
Evidence that would show the safeguards work
- E2 automation policy configuration showing limits, approvals and allowlists.
- E4 test in a staging environment that an above-threshold action is blocked or held for approval (V5).
- E3 automation action logs reconciled to approvals.
Typical UNKNOWNs
Full set of automation systems with infrastructure authority (U1; US4 on CG-15). Electrical effect of maximum credible load step from automation (U1; US4 on CG-05).
Where it is assessed
Control families: SEC-06, CMP-05, CAP-05, POW-05, THM-06. Critical gates: CG-15 Integrated digital authority over infrastructure; CG-09 Failure containment; CG-05 Resilient electrical supply. Required functions: RF-SEC-03, RF-CMP-05, RF-CMP-06, RF-CAP-01, RF-POW-06, RF-THM-08, RF-STO-03.
X19 Colocation density increase silently consumes resilient capacity
CAPDEPTHMCMPGOVCG-03CG-06CG-13
Starts with. A colocation tenant replaces equipment with a higher-density accelerator generation within its contracted power allocation, but with a less diversified, more synchronous load profile and higher liquid-cooling demand.
Service outcomes affected. Tenant training throughput; provider service levels; contractual commitments.
Propagation path (6 steps)
- CAPTenant load-profile change (DEMAND_SURGE in peak terms) CONTRIBUTES_TO loss of provider headroom (M14 on RF-CAP-02), LATENT because contracted kW are not exceeded.
- DEPResponsibility gap: neither party analyses the combined effect (M17 on RF-DEP-02).
- THMProvider takes one chiller out for maintenance assuming N+1 exists (ENABLING; HUMAN_ACTION).
- THMAt peak load, facility water supply temperature exceeds the tenant CDU's band (M07 on RF-THM-04).
- THM → CMPTenant accelerators throttle; jobs slow or fail.
- GOV → DEPContractual dispute over responsibility; evidence missing on both sides, producing evidence and attribution loss.
Safeguards, evidence and unknowns
Preconditions
The provider's capacity model assumes diversity across tenants. The tenant's CDUs depend on provider facility water within a supply-temperature band. The shared-responsibility matrix does not specify notification of load-profile change or of provider plant maintenance. The tenant has no evidence rights to provider plant telemetry.
Operating states
NORMAL (apparent) → MAINTENANCE (provider chiller maintenance) → DEGRADED → EMERGENCY.
Safeguards that should interrupt it
- Cause side: shared-responsibility record that requires notification and joint assessment of load-profile changes; provider capacity model using measured peak and correlation (interrupts steps 1–2).
- Escalation: provider maintenance readiness check against measured load; notification to tenants (interrupts step 3).
- Detection: tenant monitoring of facility water supply temperature at CDU inlet with alerts; shared telemetry (interrupts step 4).
Evidence that would show the safeguards work
- E1 shared-responsibility record and contract clauses for notification and evidence rights.
- E3 measured peak load and supply-temperature data at CDU inlets.
- E7 provider resilient-capacity calculation with measured tenant profiles.
Typical UNKNOWNs
Provider resilient capacity (U2; US4 on CG-06). Other tenants' load profiles (U2; US3).
Where it is assessed
Control families: DEP-05, SUP-02, CAP-04, THM-01, OPS-06. Critical gates: CG-03 Critical dependency visibility; CG-06 Thermal continuity; CG-13 Responsibility and authority. Required functions: RF-CAP-02, RF-DEP-02, RF-THM-04, RF-THM-03, RF-GOV-02, RF-EVA-01.
X20 Compound regional event: flood, extended grid loss, telecom loss and fuel logistics
PHYENVPOWDEPSUPNETOPSRESCG-03CG-05CG-10CG-04
Starts with. A regional storm causes river flooding, extended grid outage, loss of public telecom and road closures around the site.
Service outcomes affected. Availability; recovery; safety (flood and fuel handling); strategic or sovereignty outcomes if workloads must move.
Propagation path (7 steps)
- PHY → ENVFlood threatens switchgear and fuel-system equipment (EXTERNAL_EVENT).
- POWGrid loss (EXTERNAL_EVENT, CORRELATES_WITH step 1) TRIGGERS generator operation for an extended period.
- POWExtended running reveals fuel contamination or filter blockage (LATENT; M21 on RF-POW-02); one or more generators fail; K_OF_N redundancy at risk.
- DEP → SUPRoad closures and regional demand prevent fuel replenishment (M10 on RF-SUP-02 and RF-DEP-02; DELAYS replenishment beyond autonomy).
- NETPublic telecom loss removes WAN and remote support (links to X09, X15).
- OPSStaff fatigue and inability to relieve shifts (HUMAN_ACTION contributors).
- RESDecision to degrade to minimum Service Outcome, migrate workloads or execute a controlled shutdown before fuel exhaustion; if not pre-decided, decision delay increases the probability of uncontrolled shutdown (M23).
Safeguards, evidence and unknowns
Preconditions
Ground-level switchgear and fuel tank fill points lie within a credible flood extent; fuel autonomy is sized for a delivery interval that assumes open roads; fuel suppliers are regional and share the same roads; staff relief depends on public transport; generators have not run for an extended duration recently; fuel quality has not been tested for some time.
Operating states
NORMAL → EMERGENCY → DEGRADED (on generators, extended) → possible SHUTDOWN (controlled) → RECOVERY.
Safeguards that should interrupt it
- Cause side: flood protection of critical plant above the credible flood level; site hazard assessment (interrupts step 1).
- Escalation: fuel quality testing and polishing programme; periodic extended-duration generator runs (interrupts step 3).
- Escalation: fuel autonomy sized for the replenishment scenario, multiple suppliers with priority contracts and independent routes (interrupts step 4).
- Consequence side: pre-approved crisis priorities, graceful degradation and controlled shutdown plan; staff welfare and accommodation plan (limits steps 6–7).
- Consequence side: alternate site capacity with tested migration (limits step 7).
Evidence that would show the safeguards work
- E7 site flood assessment and plant elevation survey.
- E4 extended-duration generator run records and fuel quality test results.
- E1 fuel supply contracts, autonomy calculation and replenishment plan; E4 crisis exercise (V4).
- E4 workload migration test to alternate capacity (V6).
Typical UNKNOWNs
Fuel supplier performance under regional event (U2; US4 on CG-05). Credible flood extent under updated projections (U9; US3–US4). Staff availability (U5; US3).
## Scenario prioritisation
Scenarios shall be prioritised for analysis depth and validation effort when any of the following apply; each criterion is linked to the gate it most directly engages.
| **Priority criterion** | **Primary gate** |
|------------------------|------------------|
| Life safety or hazardous energy | CG-04 |
| Essential cooling loss | CG-06 |
| Standby or resilient power loss | CG-05 |
| Containment failure | CG-09 |
| Large capability concentration | CG-03 |
| Checkpoint or recovery-baseline corruption | CG-10 |
| Common-mode defeat of redundancy | CG-09 |
| Recovery objective at risk | CG-10 |
| Weakly evidenced external dependency | CG-03 |
| Long-lead or irreplaceable component | CG-10 |
| Control-plane or remote-access exposure | CG-15 |
| Sovereignty impact | CG-01 |
| Critical UNKNOWN (US4 or US5) | CG-12 |
Where it is assessed
Control families: PHY-01, POW-02, SUP-05, RES-03, RES-06, ENV-06, OPS-01. Critical gates: CG-03 Critical dependency visibility; CG-05 Resilient electrical supply; CG-10 Recovery readiness; CG-04 Life safety and hazardous energy. Required functions: RF-PHY-05, RF-POW-02, RF-SUP-02, RF-DEP-02, RF-NET-03, RF-OPS-05, RF-RES-02, RF-RES-05, RF-GOV-01.