WhiteBox Networks

WBN capabilities · Engineering guide

Engineer Ethernet for the collective.

Clos topology, physical wiring and transport behavior designed together for large-scale AI infrastructure.

WBN engineering 2 November 2026 Sources
  1. Workload Collectives + placement
  2. Topology Radix + rails + failure capacity
  3. Transport Queues + feedback + paths
  4. Acceptance Job progress + fault tests
The design connects application traffic to the network's behavior in both healthy and degraded states.
In this guide

Start with the work the fabric must carry.

We engineer large-scale Ethernet Clos fabrics as an alternative to InfiniBand, starting with GPU topology, job placement and collective traffic. AllReduce, AllGather, ReduceScatter and AllToAll stress the network differently. Message size, parallelism strategy, simultaneous jobs and storage traffic all influence which links become limiting.

The design brief records how GPUs reach NICs, which ranks communicate and what growth must preserve. It also separates compute, storage and management requirements. A fabric designed around aggregate switch throughput alone misses the paths that determine job completion. We connect those workload assumptions to a topology and acceptance plan that can be tested.

Count capacity at every cut, including failures.

Switch radix, port breakout and topology depth determine how much usable connectivity a Clos provides. We count host-facing and fabric-facing ports, physical modules, optical circuits and cable assemblies separately. Oversubscription is evaluated at the relevant network cuts, with expansion ports reserved deliberately.

Rail layouts and planes must match the system and collective strategy. NVIDIA's GB300 reference architecture is one concrete example: corresponding GPU positions connect through rails, with traffic separated across two planes. Its wiring should not be copied without its endpoint and topology assumptions. NVL72 network architecture ↗

We calculate surviving capacity after link, leaf, spine or plane loss. Two planes do not preserve their combined healthy bandwidth when one disappears, and physical separation does not by itself establish application failover.

Make RoCE an end-to-end operating policy.

RoCE engineering spans NIC behavior and every switch hop. We define traffic classification, DSCP or PCP trust, priority-to-queue mapping, MTU, ECN marking, sender congestion response, congestion-notification traffic and selective PFC together. Buffer allocation and pause headroom must reflect the actual rates, delays and implementation.

ECN feedback and PFC serve different purposes. Feedback aims to control offered load; priority pause can stop a traffic class when its buffer conditions require it. Their interaction, propagation and recovery need measurement. Arista's deployment guide separates RoCE data, CNPs, control and best-effort traffic, with deployment-specific thresholds rather than a universal recipe. AI fabric deployment guidance ↗

We turn that policy into versioned configuration intent for the selected NIC firmware and switch operating system, then verify the behavior under contention.

Match path selection to transport behavior.

ECMP, flowlet switching and per-packet spraying are different design choices. Large low-entropy flows can collide on one path while capacity elsewhere remains idle. Changing the distribution mechanism also changes the packet-ordering conditions seen by the receiver.

Meta's production RoCE account describes QP-aware hashing and collective-library changes, illustrating why network and application behavior must be evaluated together. NVIDIA's Spectrum-X explanation similarly ties per-packet adaptive routing to endpoint support for out-of-order arrival. Meta's deployment experience · Transport-aware adaptive routing

We assess path-delay differences, reordering and recovery support before selecting a balancing mode. Extra paths do not remove an incast bottleneck at one receiver; congestion control and path distribution remain separate parts of the design.

Evaluate Ultra Ethernet as a transport choice.

Ultra Ethernet Transport, or UET, is distinct from RoCEv2. Its transport mechanisms and software integration must be considered alongside the switching fabric. UEC's current specification is version 1.1, released in October 2026; its release notes include a verbs interface and programmable congestion management. Specification history · Version 1.1 changes

We build a support matrix around the exact transport and profile, required RDMA operations, NIC and firmware, switch behavior, congestion signaling and collective-library integration. Packet spraying, receiver reordering and interoperability must be supported by that combination. A roadmap label cannot answer those questions.

This allows RoCE and UET options to be compared against the deployment's workload and operating requirements, with a clear path from available capability to tested behavior.

Deliver a design that can be measured.

Our engineering outputs connect the traffic model, healthy and degraded capacity, port-conserving topology, cable schedule, configuration intent and acceptance plan. Validation exercises representative collectives across message sizes and job scales, then the intended training or inference workload, including competing jobs and injected failures.

The operating baseline joins job progress and completion-time tails to queue occupancy, ECN/CNP activity, PFC duration, drops, retransmissions, path use and physical-link errors. NVIDIA's telemetry example shows why a degraded physical port and transport retries need to be correlated with training performance. Workload-to-link telemetry ↗

Bring the GPU and NIC inventory, growth target and representative traffic into the endpoint worksheet. We can develop the topology, physical connection plan and testable operating design from that common starting point.

Sources and applicability

WBN AI Ethernet architecture and implementation planning. Transport behavior and acceptance targets are tied to the actual NIC, switch, firmware and workload.

  1. NVIDIA NVL72 network logical architecture

    Reference rail and plane arrangement; topology assumptions are specific to that architecture.

  2. Arista AI Network Fabric Deployment Guide

    Traffic classes, ECN, CNP handling and PFC deployment considerations.

  3. Meta: RoCE networks for distributed AI training at scale

    Production experience with topology, hashing, congestion behavior and collective-library changes.

  4. NVIDIA Spectrum-X networking platform

    Platform-specific adaptive routing, endpoint reordering and congestion-control behavior.

  5. Ultra Ethernet specification history

    Version 1.1 release dated 12 October 2026.

  6. Ultra Ethernet 1.1 release notes

    Published additions including verbs and programmable congestion management; individual implementations require their own support matrix.

  7. NVIDIA AI factory telemetry

    Correlating training performance, retransmissions and physical-port errors.