RES Resilience, Continuity and Recovery

The capability owner shall define and approve, for each in-scope Capability Instance, the minimum service outcome to be sustained under DEGRADED and EMERGENCY states and the impact tolerance, expressed as maximum tolerable duration of disruption, minimum retained capacity (for example accelerator-hours or tokens per second), maximum tolerable lost training progress, recovery time and recovery point objectives, and the business-impact basis for each. The resilience owner shall ensure every RES-02 to RES-05 control and every NET, POW, THM, CMP and STO recovery claim references these tolerances.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: NORMAL, DEGRADED, EMERGENCY, RECOVERY.

The resilience owner shall define graceful-degradation modes for each Capability Instance, stating the trigger condition (for example loss of one cooling string, utility loss on UPS autonomy, loss of a fabric plane), the action (power capping, job preemption by priority, checkpoint-and-drain, inference replica reduction, admission control), the owner of the decision and the expected residual capacity. The platform operations lead shall implement the actions as tested automation or procedures and shall demonstrate by test or analysed event that each mode achieves the expected residual capacity within the time available before the triggering constraint is exceeded.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: DEGRADED, EMERGENCY, TRANSITION.

The resilience owner shall maintain a scenario library for each site and Capability Instance that includes, at minimum, the applicable AIDCF-004 cross-domain scenarios (X01 to X20), site-specific hazards, and maintenance-induced and recovery-induced scenarios; each scenario shall have an initiating event, initial operating state, propagation path, safeguards, residual capability and recovery path. The resilience owner shall prioritise scenarios using the AIDCF-004 prioritisation criteria and shall review the library annually and after major incidents.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: NORMAL, DEGRADED, MAINTENANCE, EMERGENCY, RECOVERY, TRANSITION.

The executive sponsor shall appoint an accountable resilience owner for each site or Capability Instance with authority to set resilience requirements, approve accepted limitations and direct cross-domain testing. The resilience owner shall maintain a resilience responsibility matrix naming the responsible owner for power, cooling, network, compute, storage, control-plane and provider elements of each resilience and recovery capability, including colocation and cloud provider responsibilities.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: NORMAL, EMERGENCY, RECOVERY.

The resilience owner shall maintain a site fault-domain model that defines each fault domain (for example electrical distribution paths, UPS systems, cooling strings and CDU groups, fire compartments, water zones, network planes, compute pods, storage failure domains, control-plane clusters) with its members, and records which domains overlap. The resilience owner shall reconcile the model to as-built evidence (single-line diagrams, P&IDs, rack power and cooling mapping, fabric topology, scheduler and storage placement) at commissioning and after material change, and shall flag every redundancy claim whose members share a fault domain.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE, DEGRADED.

The resilience owner shall identify each containment boundary relied upon (for example fire compartment, electrical protection discrimination point, cooling-loop isolation valves, fabric partition, security zone, scheduler placement constraint, storage failure domain) together with the isolating device or mechanism and its required response. The domain owners shall confirm that each mechanism is installed, configured and maintained as designed and shall record mechanisms in bypass or override as temporary configurations under OPS-019.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE, DEGRADED.

The resilience owner shall perform and maintain a blast-radius analysis for each prioritised scenario and each fault domain, stating the affected assets, the affected Capability Instances, the capability consequence in outcome units and comparison with impact tolerance, including common-cause and common-mode failures (for example firmware rollout, shared controller, shared identity provider). The resilience owner shall record every blast radius exceeding tolerance as a Finding with a treatment plan or approved limitation.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E7 Derived and analytical. Optional: E3, E4, E5. Operating states: NORMAL, DEGRADED, MAINTENANCE, FAILED.

The resilience owner shall maintain an isolation validation plan that assigns a validation method from the AIDCF-004 safe validation hierarchy (V0–V7) to each containment boundary, with the highest safe level for gate-critical boundaries, and shall execute the plan over a defined cycle. The domain owners shall record results, including failed or partial isolation, and shall retest after remediation.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE, FAILED, EMERGENCY.

The resilience owner shall maintain an approved continuity strategy for each Capability Instance that selects and justifies the continuity approach (for example in-site redundancy, active-active multi-site inference, warm standby training capacity, checkpoint replication to an alternate site, contracted cloud burst, or accepted suspension), states the dependencies of the chosen approach, and is consistent with the impact tolerances in RES-001.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: EMERGENCY, FAILED, RECOVERY.

The capability owner shall identify the alternate capacity relied upon (alternate cluster, partitioned reserve, alternate site, cloud contract) with its size, compatibility (accelerator generation, software stack, fabric, storage access) and activation time, and shall secure it by reservation or contract. The platform operations lead shall demonstrate by test at a defined interval that a representative workload can be relocated to alternate capacity and resumed from checkpoint or served within the activation time.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: EMERGENCY, RECOVERY, FAILED.

The resilience owner shall obtain, for each site operator and critical provider (colocation, utility, carrier, cloud, OEM support), evidence of their continuity arrangements relevant to the outcome (for example operator emergency response, fuel contracts, carrier restoration commitments, provider recovery objectives), and shall reconcile provider objectives with the Capability Instance's tolerances. The resilience owner shall record every mismatch as a dependency limitation.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E5 Independent and third-party. Optional: E3, E6. Operating states: EMERGENCY, FAILED, RECOVERY.

The capability owner shall approve a priority scheme stating, for scarce power, cooling, compute and network capacity during DEGRADED, EMERGENCY and RECOVERY states, which workloads, tenants and services are protected, preempted and restored first, consistent with contractual commitments and life-safety precedence. The platform operations lead shall implement the scheme in scheduler priority classes and runbooks.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: DEGRADED, EMERGENCY, RECOVERY.

The recovery manager shall maintain a recovery dependency map for each Capability Instance, derived from DEP-004, listing every prerequisite system, service, credential, artefact, person and provider required to restore the outcome from a cold or failed state, and shall identify circular dependencies (for example identity provider hosted on the cluster it unlocks, DNS required to reach the time source, BMC firmware repository on the failed storage) and the means by which each is broken. The recovery manager shall verify the map against the last integrated recovery test.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: FAILED, RECOVERY, SHUTDOWN, EMERGENCY.

The recovery manager shall maintain a black-start and restoration sequence for each site and Capability Instance that orders restoration through power (utility or generation, distribution, UPS), cooling (heat rejection, facility water, CDUs and liquid loops before IT load), out-of-band and core network, time and identity services, storage (metadata and quorum before data services), compute (BMC, firmware validation, health checks, burn-in) and AI control plane (cluster manager, scheduler), with entry criteria, hold points, load-step limits and safety checks at each stage. The recovery manager shall demonstrate the sequence by integrated test or exercise at a defined interval and shall record achieved stage durations against the recovery time objective.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: SHUTDOWN, FAILED, RECOVERY, TRANSITION, EMERGENCY.

The recovery manager shall ensure that configuration backups of network devices, controllers, BMS/EPMS, CDU controllers, storage controllers, cluster managers and identity systems are taken at a defined interval, are integrity-protected (hashed or signed), are stored outside the failure and compromise domain they protect, and are tagged with the approved baseline they correspond to. The recovery manager shall test restoration of a sample of each system class at a defined interval, including verification that the restored configuration matches the approved baseline and not a compromised or drifted state.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: RECOVERY, FAILED, MAINTENANCE.

The capability owner shall set a checkpoint policy per training Capability Instance stating checkpoint interval (justified against failure rate, checkpoint cost and maximum lost work, for example using a Young/Daly-style optimisation), checkpoint retention, and the number and placement of durable copies outside the training cluster's failure domain. The platform operations lead shall verify checkpoint integrity at write and before restore, and shall test at a defined interval that a representative job can be restarted from checkpoint (including on a different set of nodes) and reaches equivalent training state within RTO and lost-work tolerance.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, FAILED, RECOVERY, DEGRADED.

The resilience owner shall maintain a component test programme listing each resilience-relevant component class, the test method, frequency and acceptance criteria, coordinated with domain maintenance programmes. Domain owners shall execute tests per programme, record results, and raise and close defects; the resilience owner shall review programme adherence quarterly.

Evidence required: E3 Operational and observational; E4 Validation and test. Optional: E1, E2. Operating states: NORMAL, MAINTENANCE.

The resilience owner shall plan and execute integrated scenario tests for prioritised scenarios from RES-003 (for example utility loss with generator start and UPS transfer under AI load, cooling string loss at sustained load, fabric plane loss during checkpoint, control-plane loss), measuring capability-level outcomes (job survival, lost work, inference availability, restoration time) as well as system behaviour. The resilience owner shall define test objectives, safety case, abort criteria and success criteria tied to impact tolerances before each test and shall record results and defects.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, EMERGENCY, FAILED, RECOVERY, TRANSITION.

The resilience owner shall run tabletop exercises on a defined cycle covering scenarios not testable live, including at least one cyber-physical scenario (for example BMS/EPMS compromise, X05) and one compound event (X20) per cycle, with participation of crisis command, facility, IT, security and key providers. The resilience owner shall record decisions, timings, gaps and actions and shall track actions to closure.

Evidence required: E3 Operational and observational; E6 Human and experiential. Optional: E1, E4. Operating states: EMERGENCY, RECOVERY, FAILED.

The resilience owner shall maintain a lessons register capturing every defect and lesson from tests, exercises and incidents with owner, due date, and the plan, runbook, design or control affected. The resilience owner shall require retest or equivalent verification before closure of any lesson affecting a gate-critical safeguard and shall report overdue items to governance monthly.

Evidence required: E1 Documentary and declarative; E3 Operational and observational. Optional: E4, E7. Operating states: NORMAL, RECOVERY.

The executive sponsor shall establish a crisis command structure with strategic, tactical and operational levels, named primary and alternate role holders, activation criteria, decision authority (including authority to shed load, suspend tenants, invoke alternate capacity and order controlled shutdown), and command facilities and communication channels that remain available when site systems, corporate identity or primary communications fail. The crisis management lead shall demonstrate activation and decision timings in exercises or real events.

Evidence required: E1 Documentary and declarative; E3 Operational and observational; E4 Validation and test; E6 Human and experiential. Optional: E5. Operating states: EMERGENCY, FAILED, RECOVERY.

The crisis management lead shall maintain a communications plan listing stakeholders, notification obligations and deadlines (contractual and regulatory), approved message templates, approval authority and channels that remain available during site and identity failures. The crisis management lead shall record each notification issued during events and exercises with timestamp and approver and shall review timeliness against obligations.

Evidence required: E1 Documentary and declarative; E3 Operational and observational. Optional: E6. Operating states: EMERGENCY, RECOVERY.

The change authority shall define an emergency change procedure stating who may authorise emergency changes to infrastructure, OT and control-plane systems, the minimum record to be captured at the time, and the deadline for retrospective review. The recovery manager shall ensure every emergency change is retrospectively reviewed, either reverted or formalised, and that baselines and evidence are updated, and shall report unreconciled emergency changes.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E6. Operating states: EMERGENCY, RECOVERY.

The resilience owner shall require a post-event review within a defined time after every material event, covering timeline, root and contributing causes, safeguard performance against expectation, recovery performance against tolerance, and required adaptations to design, operations, controls and evidence. The resilience owner shall raise reassessment triggers for affected controls and gate results and shall track adaptations to completion.

Evidence required: E1 Documentary and declarative; E3 Operational and observational; E7 Derived and analytical. Optional: E6. Operating states: RECOVERY, NORMAL.