NET AI Network and Interconnect Architecture

The network architecture owner shall maintain an approved topology design for every back-end (scale-out) fabric and scale-up domain in scope, stating the topology class (for example rail-optimised multi-plane, two-tier leaf-spine or three-tier fat-tree/Clos, or rack-scale scale-up domain), switch radix, tier count, rail and plane assignment, endpoint-to-leaf mapping and the design bisection bandwidth per Capability Instance. The fabric operations lead shall reconcile the as-built fabric (LLDP or subnet-manager discovered topology, cabling records and port maps) to the approved design at commissioning and after every material change, and shall record every deviation as either an approved variance or a Finding.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE.

The network architecture owner shall define, for each redundant path group (spine uplinks between halls, storage and front-end uplinks, WAN and data-centre interconnect circuits, OOB uplinks), the required diversity attributes: separate cable routes and containment, separate building entries, separate carriers or carrier-neutral dark-fibre routes, and separate optical line systems where DWDM is used. The connectivity manager shall obtain route evidence (carrier KMZ or route maps, cable-route surveys, containment records) for each path and shall record every shared segment, shared conduit, shared meet-me room or shared upstream carrier as a common-mode dependency.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE, DEGRADED.

The network architecture owner shall state the design oversubscription ratio at every tier boundary (leaf-to-spine, spine-to-superspine, pod-to-core, storage and front-end uplinks) and shall justify each ratio against the traffic matrix of the declared workloads, including all-reduce, all-to-all (for example expert-parallel) and checkpoint bursts. The fabric operations lead shall re-derive effective oversubscription after link failures, partial builds and planned maintenance, and shall record the minimum tolerated effective ratio for each Service Outcome.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: NORMAL, DEGRADED, MAINTENANCE, TRANSITION.

The network architecture owner shall define the fault domains of each fabric, including each switch, rail, plane, pod, spine group, subnet manager or fabric controller instance, and the power and cooling zones serving fabric equipment, and shall state for each the maximum affected endpoints and the resulting outcome impact. The fabric operations lead shall demonstrate, by controlled test or by analysed incident, that loss of a leaf, a spine, a rail or plane, a subnet-manager master and one power feed to a switch row each stay within the declared blast radius, and that fabric equipment does not share an undisclosed power or cooling fault domain with its own redundancy.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE, FAILED, RECOVERY.

The fabric operations lead shall establish and maintain latency baselines for each path class (intra-leaf, intra-pod, cross-spine, cross-plane, storage path, front-end path) including idle round-trip time, loaded tail latency at defined percentiles and switch-hop count, measured with a repeatable method (for example perftest-class RDMA latency tools or active probing). The fabric operations lead shall alert on deviation from baseline beyond a defined threshold and shall re-baseline after any material NOS, firmware, topology or congestion-control change.

Evidence required: E2 Configuration and technical state; E3 Operational and observational. Optional: E1, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE.

The fabric operations lead shall monitor, per port and per traffic class, pre-FEC and post-FEC bit error indicators, FEC corrected and uncorrectable codeword counts, CRC/symbol errors, link flaps, discards, and RDMA transport retransmission and out-of-sequence counters at NICs, against defined thresholds. The fabric operations lead shall automatically isolate or drain links that exceed uncorrectable-error or flap thresholds, shall track each to closure, and shall verify that the lossless traffic class shows no congestion drops under normal operation.

Evidence required: E2 Configuration and technical state; E3 Operational and observational. Optional: E1, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE.

The network architecture owner shall define a congestion-control design per fabric stating traffic-class mapping, ECN marking thresholds, PFC enablement and headroom per priority, PFC watchdog behaviour, end-host rate-control algorithm and parameters, and adaptive routing settings, and shall analyse the design for cyclic buffer dependency (PFC deadlock) and pause-storm propagation. The fabric operations lead shall verify by controlled test that incast and congestion scenarios representative of declared workloads do not cause lossless-class drops, sustained pause propagation beyond the congested port, or deadlock, and that the PFC watchdog detects and breaks a stuck-pause condition within the configured time.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE.

The fabric operations lead shall run a defined collective benchmark suite (for example NCCL- or RCCL-class tests and a representative workload proxy) at commissioning, after every material fabric, NIC or library change and at a defined periodic interval, covering message sizes and scales used by declared workloads, and shall compare achieved bus bandwidth and latency to acceptance targets. The capability owner shall set the acceptance targets from the performance Service Outcome and shall accept or reject results for degraded scale (for example one plane or one spine unavailable).

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, TRANSITION.

The network architecture owner shall define the storage network design: whether storage traffic uses a dedicated fabric, a dedicated plane or a separate traffic class on the compute fabric; the path from compute nodes to storage servers and metadata services; capacity per storage server; and the traffic-class and congestion policy that prevents checkpoint bursts from degrading collective traffic. The storage platform owner and fabric operations lead shall jointly test a synchronised checkpoint write and a dataset-read burst at declared scale and confirm achieved throughput and the effect on concurrent collective performance meet targets.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED.

The network architecture owner shall maintain an approved front-end network design covering node attachment (dual-homed or single-homed), routing protocol and ECMP design, uplinks to core and edge, load-balancer and gateway placement, and capacity for inference request and data-ingest peaks. The network operations manager shall reconcile as-built configuration and attachment to the design at commissioning and after material changes, and shall confirm that loss of a single front-end switch or uplink does not disconnect inference endpoints or orchestration control traffic beyond tolerance.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, DEGRADED, MAINTENANCE.

The network operations manager shall operate each service gateway cluster (load balancers, inference API gateways, NAT, edge routers and in-path firewalls) as a redundant pair or cluster with health checks, session-state handling and capacity monitoring, and shall define capacity thresholds for connections, throughput and TLS handshakes. The network operations manager shall review gateway capacity and failover events monthly and shall demonstrate that gateway failover occurred, or was tested, within tolerance in the assessment period.

Evidence required: E2 Configuration and technical state; E3 Operational and observational. Optional: E1, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE.

The network architecture owner shall define the isolation model for shared fabrics: InfiniBand partition keys, VRFs, VXLAN/EVPN segments, VLANs, RDMA queue-pair and memory-region protections, traffic-class guarantees and rate limits. The fabric operations lead shall verify by test that endpoints in one isolation domain cannot reach endpoints in another, and that a saturating or misbehaving tenant cannot reduce another tenant's guaranteed bandwidth or trigger fabric-wide pause beyond the defined tolerance.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED.

The network architecture owner shall define the management network architecture, separating in-band management, BMC/out-of-band management and facility OT management, and shall prohibit management interfaces from being reachable from tenant or data networks except through defined conduits. The network operations manager shall reconcile the management network attachment of every switch management port, BMC, PDU, console server and storage controller to the design, and shall record any device whose management is only reachable in-band as a Finding.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE.

The network operations manager shall provide serial console access to every switch, router, firewall and storage controller, and BMC/KVM access to every server, through console servers or OOB infrastructure that is independent of the data fabric, and shall require individual authentication, session logging and role-based authorisation for console sessions. The network operations manager shall verify console reachability for every device at a defined interval and after rack moves.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE, FAILED, RECOVERY.

The network architecture owner shall design the OOB network with its own switching, power supply from a protected source with defined autonomy, independent upstream connectivity (for example a separate carrier or cellular path), and locally available authentication and name resolution, and shall document every dependency the OOB network has on the systems it is used to recover. The network operations manager shall test at a defined interval that OOB access to critical devices is achievable with the production fabric, primary WAN and primary identity provider unavailable.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, FAILED, EMERGENCY, RECOVERY.

The network operations manager shall provision break-glass local credentials or offline authentication for network devices, fabric controllers, subnet managers and console servers, stored in a sealed or vaulted mechanism with dual control, and shall test break-glass login on a sample of device classes at a defined interval. The infrastructure security owner shall review every break-glass use within a defined time, rotate the credentials afterwards, and confirm that break-glass accounts are disabled for routine use.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, EMERGENCY, FAILED, RECOVERY.

The cabling plant owner shall maintain a cabling design standard for the site covering pathway and containment fill limits, segregation from power, minimum bend radius by cable type, MPO/MTP polarity method, connector cleanliness acceptance, labelling scheme and test acceptance (insertion loss and, where applicable, reflectance), aligned with recognised data-centre cabling standards. The cabling plant owner shall approve deviations, and installers shall be bound to the standard contractually.

Evidence required: E1 Documentary and declarative; E7 Derived and analytical. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE.

The network architecture owner shall maintain a qualified-optics matrix listing approved transceiver and cable part numbers, firmware versions, host switch/NIC models and NOS/firmware versions, reach and link-budget class. The fabric operations lead shall reconcile installed optics (from digital diagnostic and CMIS inventory) to the matrix, shall monitor optical diagnostics (Rx/Tx power, temperature, bias, pre-FEC error indicators), and shall quarantine unqualified or degrading optics.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE.

The network operations manager shall require an approved work order for every physical patch or cross-connect change, specifying both ends, labels, affected path groups and redundancy impact, and shall verify after each change that the link came up on the expected ports and that the cabling records were updated. The network operations manager shall prohibit simultaneous work on members of the same redundant path group unless explicitly approved.

Evidence required: E2 Configuration and technical state; E3 Operational and observational. Optional: E1, E7. Operating states: NORMAL, MAINTENANCE.

The network operations manager shall maintain a path-documentation system (DCIM or cable-management database) recording every in-scope link from endpoint port through patch panels, trunks and cross-connects to the far-end port, with labels matching the physical plant and links to circuit IDs for external paths. The network operations manager shall audit a sample of paths physically and electronically at a defined interval and shall correct discrepancies, with an accuracy target set as a capability-profile parameter.

Evidence required: E1 Documentary and declarative; E2 Configuration and technical state; E3 Operational and observational. Optional: E5, E6. Operating states: NORMAL, MAINTENANCE, RECOVERY.

The network architecture owner shall deploy fabric control functions with redundancy (master/standby subnet managers, clustered controllers, redundant route reflectors) placed in separate power and rack fault domains, and shall document the data-plane behaviour when the control function is lost (forwarding continues with stale state, or fails). The fabric operations lead shall test controller and subnet-manager failover at a defined interval and after upgrades and shall confirm forwarding continuity and failover time within tolerance.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, FAILED, RECOVERY.

The fabric operations lead shall maintain a version-controlled source of truth for intended network configuration and an approved baseline of NOS, switch firmware, NIC firmware and driver, optics firmware and controller versions; shall detect configuration and version drift at a defined interval; and shall roll out changes in stages (canary, partial, full) with automated pre- and post-change checks and defined rollback. The network operations manager shall ensure that no single change can be applied simultaneously to both members of a redundant pair or to all planes of a fabric.

Evidence required: E2 Configuration and technical state; E3 Operational and observational. Optional: E1, E7. Operating states: NORMAL, MAINTENANCE, TRANSITION, RECOVERY.

The fabric operations lead shall collect streaming or polled telemetry from all in-scope switches and NICs covering interface counters, error counters, PFC pause and ECN mark counts, buffer occupancy, and flow sampling, at a resolution sufficient to observe collective-scale events, and shall retain it for the evidence retention period. The fabric operations lead shall monitor telemetry pipeline health and alert on collection gaps.

Evidence required: E2 Configuration and technical state; E3 Operational and observational. Optional: E1, E7. Operating states: NORMAL, DEGRADED, MAINTENANCE, RECOVERY.

The network architecture owner shall deploy time sources (with at least two independent references such as GNSS and a holdover oscillator or a remote time service), DNS, DHCP, IPAM and network AAA as redundant services placed in separate fault domains, and shall document every circular dependency between these services and the systems they support. The network operations manager shall monitor time offset, stratum and source agreement, and service availability, and shall test failure of a time source and of a DNS/AAA instance at a defined interval.

Evidence required: E2 Configuration and technical state; E3 Operational and observational; E4 Validation and test. Optional: E5, E7. Operating states: NORMAL, DEGRADED, FAILED, RECOVERY.