Zephyr USB networking under overload: NAK or drop?

30 October 2026, updated 31 October 2026

In the previous post we measured a stall in the Zephyr USB network drivers on three boards and six emulated CPUs. The fix for CDC-NCM (#123515) is merged: when the driver cannot get a receive packet, it now drops the data instead of waiting. This post asks whether drop is the right policy. USB has a third option: the device can answer the PC with NAK ("not now"), and the PC sends the data again later. When a small USB network device receives more than it can process, should it drop, or should it NAK?

We measured both on three boards, with dense rate steps and repeated runs. The data shows what NAK buys, what it costs, and where the loss goes in each case.

Three options when the receive pool is empty

A USB network function (CDC-NCM or CDC-ECM) receives each Ethernet frame from the PC in a bulk OUT transfer. In the transfer-complete callback, the driver allocates a network packet for the frame. In Zephyr, this takes one packet descriptor from the RX packet slab (CONFIG_NET_PKT_RX_COUNT, 4 to 8 in our builds) and data fragments from the RX data pool (CONFIG_NET_BUF_RX_COUNT × 128 bytes, 16 to 32). The application frees both only when it has read the datagram. If either is empty, the driver has three choices:

Wait (K_FOREVER, the code as found). The driver blocks until a packet is free. While it blocks, it does not give the controller a new receive buffer, so the USB hardware answers every OUT from the PC with NAK. The PC holds the data and tries again. This is flow control, but the wait has no time limit, and it runs in the only thread of the USB stack (see below).
Drop (K_NO_WAIT, the merged NCM fix and our ECM fix). The driver discards the frame and gives the hardware a new buffer at once. The device always keeps working. The data is lost, and recovery is up to the end points: TCP sends the data again; for UDP, only the application can.
Bounded defer (not implemented). The driver returns without a new receive buffer, so the hardware NAKs, but it tries again later and drops after a time limit. This keeps the flow control of "wait" and the liveness of "drop", at the cost of more code.

We have not built the bounded defer. But the old CDC-ECM code is a clean experiment for the NAK side: CDC-ECM handles one frame per callback, so its wait is a NAK period with no other side effect while the shortage is short. The ECM fix differs from it in one argument only (K_FOREVER → K_NO_WAIT). So "ECM, waits" against "ECM, drops" is NAK against drop on the same driver.

Caveat: we did not observe the NAK handshakes on the bus. That the controller NAKs while no receive buffer is queued follows from how the three device controllers work, and from the PC-side counters (the PC sends nothing more while the board waits), but it is not measured on the wire. A NAK counter on the Teensy (the i.MX RT ENDPTNAK register) or a bus analyser would confirm it.

Method

Each board runs a test application that reads every UDP datagram at once and sends it back (32-byte payload, 74-byte Ethernet frame). The PC offers a fixed rate for 2 s per step and counts the echoes. Rate steps are linear: 10 per second on the Pico, 30 on the STM32F3, 70 on the Teensy, up to about 3 to 4 times each board's limit, then a few steps up to 50 000 per second. Each board and variant ran in 3 independent passes; each pass flashed the firmware again. The plots show the mean and ±1 standard deviation over the passes.

Goodput is the echo rate between 0.5 s and 2 s into each step. We exclude the start of the step, and the echoes that arrive after its end, because with flow control the PC queues data and delivers it late; counting those would make the result depend on the step length. (Our previous, coarser plot counted them; its values for the waiting ECM code were about 7 % too high.)

For each step we also account for every datagram that did not come back. It was lost in one of three places:

Refused by the PC socket: send() failed with EAGAIN. The sender knows.
Dropped in the PC's network queue: send() succeeded, but the frame never reached the USB interface (its tx_packets counter). Linux does not report this to a UDP sender unless it sets IP_RECVERR.
Lost after the wire: the PC sent the frame to the board, and no echo came back. The board dropped it (or its reply).

Variants: A = NCM, waits (as found); B = NCM, drops (merged fix); C = NCM, drops, with a direct copy that needs one packet less per transfer; E = ECM, waits (current main); F = ECM, drops (our fix). Boards: Teensy 4.1 (i.MX RT1062, 600 MHz, high-speed USB), STM32F3DISCOVERY (72 MHz, full-speed USB), Raspberry Pi Pico (RP2040, 133 MHz, full-speed USB).

Result 1: NAK keeps the goodput; drop loses it under overload

All variants deliver every reply up to a limit: 2170 per second on the Teensy for ECM (2710 to 2870 for NCM, which packs two frames per transfer), 1290 to 1440 on the STM32F3, and 300 to 340 on the Pico. The three passes agree closely: most standard deviations are below 2 %, so the error bars are often smaller than the markers. Above the limit, the curves split:

ECM, waits (NAK) stays level at its maximum, however hard the PC pushes: about 2190 per second on the Teensy, 1330 on the STM32F3, 320 on the Pico, also at 50 000 offered.
ECM, drops peaks at the limit and then falls as the offered rate rises: on the Teensy to 730 per second at 7000 offered and 380 at 50 000; on the Pico to 170 at 1000 offered and 100 at 5000. The board spends its time on frames that it receives and then throws away. Network engineers call this receive livelock.
NCM, drops (B and C) peaks higher, because NCM carries two frames per transfer, and then falls in the same way. The direct copy (C), which needs one packet less per transfer, helps on the STM32F3 (about 9 % more under overload) and makes no clear difference on the Teensy and the Pico. The jump of the Pico NCM curves at 820 per second repeats in every pass; we do not know its cause yet.
NCM, waits (the code as found) collapses to a few replies per second just above the limit: this is the stall of the first post.

The same data on a logarithmic rate axis, up to 50 000 offered per second: the waiting ECM code stays level to the end, and the drop variants settle at a fraction of their peak.

Result 2: NAK moves the loss to the PC; drop hides it in the device

Under overload, both policies lose data; the difference is where. With NAK (ECM, waits), no datagram that reached the wire was lost on any board. The excess stays in the PC: most of it is refused by the socket, so the sender knows, and the rest is dropped in the PC's network queue. With drop, the PC accepts everything and sends it over USB, and up to 90 % of it is lost in the device, where the sender cannot see it.

This is the real argument for NAK: the decision to drop moves to the PC, which has the memory, the CPU, a queue policy, and a direct signal to the sender. The device stops spending its small CPU on data it cannot keep.

What NAK costs

Bus time at full speed. NAK is cheap on the bus only at high speed. The USB 2.0 specification (section 8.5.1) says it directly: for full- and low-speed devices, "by the time the endpoint NAKs, most of the full-/low-speed bus time for the transaction had been used", so "the full-/low-speed bus has poor utilization when there is a high frequency of NAK'd OUT transactions". The PC sends the whole data packet every time, and the device refuses it at the end. High-speed devices must support PING: the PC asks first with a short token and sends the data only when the device has space. So on the STM32F3 and the Pico, a NAKing device uses bus time that other bulk devices on the same bus (a USB disk, for example) could use. Interrupt and isochronous traffic, such as a keyboard or audio, has reserved bus time and is not affected.

Dependence on the host. How often the PC retries a NAKed transfer is not fixed by the specification. Each host controller (EHCI, xHCI) and its driver decides, and the PC never gives up because of NAK alone. The device cannot know what this costs on a given PC.

The wait is safe only while it is short. In this flood test, the application reads all the time, so each wait of the old ECM code is short: the PC's endpoint-0 probe (a GET_STATUS request every 100 ms) got every answer, in less than 1 ms on the Teensy. But the same code, with an application that reads slowly, waits for a long time, and then the whole USB device stops. That is the part of the story below.

Why a long wait stops the whole device

Diagram: an empty packet pool makes the NCM or ECM callback wait; the single usbd thread then stops every USB function of the device; the PC cancels transfers after 5 seconds; the CPU and other threads are not affected

The Zephyr USB device stack (subsys/usb/device_next) runs all its work in one thread, called usbd. This thread delivers every completed transfer to every USB function of the device, and it also answers the standard requests on endpoint 0, which the PC uses to manage the device. So when one driver waits in this thread, everything else waits too. What we measured in the previous post while the NCM driver waited for a long time (old code, slow application, three boards):

Part	Effect	Evidence
USB network interface	stops; the PC cannot send	PC transmit rate falls to zero
Standard USB requests (endpoint 0)	19 to 25 % of GET_STATUS requests got no answer within 500 ms	PC-side probe every 100 ms
USB serial console in the same device (Teensy 4.1)	output up to 8.6 s late	1 Hz heartbeat lines
The PC	the Linux TX watchdog fires after 5 s and cancels 3 to 17 transfers; data is lost	kernel log, interface counters
The CPU and other threads	not affected	a 1 ms monitor thread ran on time

The driver cannot know in advance whether a shortage will be short (a burst) or long (a slow application). So an unbounded wait is not acceptable, even though it gave the best goodput in Result 1. Nothing in the stack states or checks that a driver must not wait in this thread. A first manual survey of the new USB device stack found the same pattern in the Bluetooth HCI driver (it waits for a Bluetooth transmit buffer) and a possible case in mass storage; we will measure them before we propose fixes.

So: NAK or drop?

Drop, as the default. The reasons, in order of weight:

Liveness is decidable. With K_NO_WAIT, the question "can the USB thread stop?" has the answer "no", and one line of code shows it. With any wait, the answer depends on what every other part of the system does with network packets.
It is what the network expects. An Ethernet controller with a full receive ring drops frames. IP does not retransmit; TCP does, and it reads the loss as congestion and slows down. The USB device then behaves like every other network interface.
The benefit of NAK depends on things the device cannot control: the bus speed (cheap with PING at high speed, expensive at full speed), the host controller's retry behaviour, and other devices on the same bus.
A bounded NAK still needs a drop when its time runs out. So the drop path must exist and be correct anyway; NAK only adds a stage in front of it.

The price of drop is in Result 1 and Result 2: under sustained overload, goodput falls (receive livelock), the loss is invisible to the sender, and a TCP sender that is far away recovers slowly after each loss. Two consequences for the driver: drop as early and as cheaply as possible, before any copy or parse, and count the drops so that an operator can see them.

NAK as an optional improvement, where its conditions hold: a high-speed link (PING), a known host, and a bounded defer time followed by a drop. On our data, the waiting ECM code kept about 2180 replies per second on the Teensy at 7000 offered, where the drop fix delivered 730. Whether that is worth the extra code is a question for each product, not for the default driver.

Submit the CDC-ECM drop fix to the Zephyr project.
Finish the tree-wide check for waits in the usbd thread, and measure the Bluetooth-over-USB case before any fix.
Measure the bus cost of NAK at full speed: two boards behind one hub, one NAKing, one moving bulk data.
Make the drop cheaper and visible: drop before any copy, and count the drops.