WhiteBox Networks

AI infrastructure · Engineering guide

RoCEv2: engineer the complete traffic path.

A working Ethernet link is the starting point. RDMA performance depends on classification, congestion control, buffering and endpoint behavior.

WBN engineering 2 November 2026 Sources
  1. Sender Rate control
  2. Switch Queue + ECN
  3. Receiver CNP feedback
ECN marks travel toward the receiver; congestion-notification packets return to the sender. PFC, when used, is a separate link-local mechanism.
In this guide

What RoCEv2 carries.

RoCEv2 carries RDMA transport over UDP/IP and Ethernet. The well-known UDP destination port is 4791; the source port can provide flow entropy for network forwarding such as ECMP. Unlike RoCEv1’s Layer 2 encapsulation, RoCEv2 can traverse routed IP networks. RDMA operation still depends on compatible endpoint support and configuration. NVIDIA protocol reference ↗

UDP encapsulation does not make RoCE a TCP flow, and it does not mean every RDMA operation is an unreliable datagram. The selected RDMA transport defines completion and recovery behavior. Keep application semantics, network forwarding and link-level error handling separate when diagnosing a slow job.

ECN, CNP and PFC solve different problems.

Mechanism Where it acts What it does
ECN Congested network queue Marks eligible IP traffic to signal congestion without requiring a drop.
CNP Receiver back to sender Reports congestion to the source NIC’s congestion-control mechanism.
PFC One Ethernet link, selected priorities Pauses traffic to protect a class from buffer exhaustion.

NVIDIA’s Cumulus documentation provides both a lossless mode using ECN with PFC and a lossy mode using ECN without PFC. “RoCEv2 always requires PFC” is therefore too broad. Choose an operating model supported by the NICs, switch software and workload, and validate it as a system. Cumulus 5.11 RoCE modes ↗

A lossless configuration is an engineered behavior under stated conditions. It is not a promise that any traffic pattern, cable length or failure can never cause a drop.

Trace one packet’s classification end to end.

Write down the host’s traffic marking, the switch trust policy, the mapping into a traffic class, that class’s queue and buffer behavior, and the treatment on every egress hop. Include the return path for CNPs. A correct setting on one switch does not establish a consistent path.

Do not copy a priority number into every device and assume the meanings line up. A DSCP value, an internal switch priority, a traffic class and a PFC priority are related through explicit mappings. Keep those mappings with the hardware and software release. NVIDIA warns that generated RoCE configurations for one switch ASIC are not automatically applicable to another. Implementation applicability ↗

Also check MTU, routing, neighbor resolution and firewall treatment. Follow the NIC’s actual RDMA device and GID selection into the intended network; a successful ordinary IP ping proves only a much smaller part of that path.

Buffer headroom is a timing problem.

When a port asks its peer to pause, traffic already in flight continues to arrive. A useful first calculation is rate multiplied by reaction time. At a nominal 400 Gb/s, one microsecond represents 50,000 bytes; at 800 Gb/s, it represents 100,000 bytes.

These illustrative byte counts are not queue-size recommendations. Actual headroom must include the implementation’s signaling and reaction delays, cable propagation, packet boundaries, accounting units and shared-buffer behavior. The calculation explains why changing link speed or topology can invalidate a previously comfortable buffer setting.

Persistent pause can spread backpressure. A PFC watchdog may deliberately stop honoring pause and drop traffic to break a stuck condition; its detection and recovery behavior must be understood before treating its counters as ordinary congestion. NVIDIA PFC watchdog behavior ↗

Fix physical errors before tuning congestion.

RoCE does not require a special “RoCE wavelength.” It requires an Ethernet physical connection supported by both endpoints, plus the intended RDMA and network configuration. Confirm module recognition, lane mode, FEC and stable error behavior before interpreting transport retries as a buffer problem.

A useful fault record separates rising physical errors, packet drops, ECN marks, CNPs, pause duration and application-level stalls. Correlate counters over the same interval. A cumulative value without its starting point and elapsed time can hide whether a fault is active.

NIC recovery behavior also depends on release and capability. NVIDIA’s MLNX_OFED history documents selective retransmission support and changes in defaults for particular adapters. Treat those notes as release-specific evidence rather than assuming every NIC repairs loss in the same way. Endpoint recovery example ↗

Validate the traffic pattern that matters.

  1. One pair: establish the intended RDMA path, direction, message sizes and expected host/GPU placement.
  2. Many to one: exercise incast while observing queue occupancy, marks, pause and completion time.
  3. Concurrent jobs: measure interference and tail behavior, not just aggregate bandwidth.
  4. Mixed services: verify that storage, management and ordinary Ethernet traffic receive their intended treatment.
  5. Failure and recovery: test the approved link or path failure cases and record how the application recovers.

Save hardware, firmware, switch configuration, traffic matrix, test duration and counter deltas with the result. “Lossless enabled” is configuration evidence; a completed workload with understood behavior is operational evidence.

Sources and applicability

Original WBN engineering guidance. Protocol and implementation references checked 2 November 2026. Cumulus examples are explicitly release-specific; this page supplies no universal queue thresholds or switch configuration recipe.

  1. NVIDIA DOCA — RDMA over Converged Ethernet

    Protocol encapsulation, UDP 4791, IP routing and source-port entropy.

  2. NVIDIA Cumulus Linux 5.11 — RoCE

    A versioned implementation example for ECN/PFC modes and mapping. Its defaults are not universal recommendations.

  3. NVIDIA Cumulus Linux 5.16 — Quality of Service

    PFC watchdog, pause propagation and recovery. Behavior varies by release; 5.16.8 includes detection changes.

  4. NVIDIA MLNX_OFED 23.10-3.2.1.1 LTS — feature history

    Adapter-specific selective-repeat and congestion-counter history; verify the deployed release.