Zephyr USB CDC NCM investigation
On 26 October we described a lock-up in the Zephyr USB network driver and offered a two-line fix, pull request #123515. The Zephyr project merged it on 29 October. This post shows the evidence in detail. We tested Zephyr as we found it and with each fix, on three real boards and six emulated CPUs. We also found the same defect in the second USB network driver, and some smaller problems.
Summary
- The defect. When the network stack has no free receive packet, the USB network drivers of Zephyr (CDC-NCM and CDC-ECM) wait in the USB thread. While they wait, the whole USB device stops: the network, the USB serial console, and the answers to standard USB requests from the PC. The CPU continues to run.
- The fix. The drivers must not wait. They drop the received data, as an Ethernet driver does when its receive buffers are full.
- The evidence. On all three boards, with the old CDC-NCM code, the USB thread waited for 99.6 to 99.8 % of each 10-second test phase, and 19 to 25 % of the PC's USB control requests got no answer within 500 ms. With the fix: 0 % and 0 failed requests.
- CDC-ECM still has the defect on the current Zephyr main branch. We prepared the same one-line fix for it.
Starting point
| Item | Value |
|---|---|
| Zephyr | main branch, commit 5d2ed837a8c (29 October 2026, version v4.5.0-rc3-671) |
| Toolchain | Zephyr SDK 1.0.1 (GCC 14.3) |
| PC | Ubuntu 24.04.5, Linux 7.0, kernel drivers cdc_ncm and cdc_ether |
We built five variants from the same commit, so that only the driver code is different:
| Variant | Driver | Code | When no receive packet is free |
|---|---|---|---|
| A | CDC-NCM | main with #123515 removed (the state as we found it) | waits without a time limit |
| B | CDC-NCM | main (#123515 merged) | drops; a temporary packet uses one receive packet |
| C | CDC-NCM | main + "direct copy" patch (not yet submitted) | drops; no temporary packet |
| E | CDC-ECM | main | waits without a time limit |
| F | CDC-ECM | main + ECM patch (not yet submitted) | drops |
Devices tested
| Device | CPU | USB device controller (Zephyr driver) | Variants |
|---|---|---|---|
| Teensy 4.1 | NXP i.MX RT1062, Cortex-M7, 600 MHz | high speed, udc_mcux_ehci | A B C E F |
| STM32F3DISCOVERY | STM32F303VC, Cortex-M4F, 72 MHz, 40 KB RAM | full speed, udc_stm32 | A B C E F |
| Raspberry Pi Pico | RP2040, Cortex-M0+, 125 MHz | full speed, udc_rpi_pico | A B C E F |
| native_sim | x86 (the PC itself) | virtual USB host and device controller | A B C |
| QEMU | Cortex-M3; RISC-V 32; and big-endian SPARC (LEON3), OpenRISC 1000, MIPS (Malta) | virtual USB host and device controller | A B C |
Test procedure
Emulated: a Zephyr test with a virtual USB host
Zephyr has a virtual USB host controller and a virtual USB device controller that connect inside one firmware image. Our test uses them: a USB host in the same image sends hand-built NCM transfers to the real CDC-NCM driver. The test then checks two things:
- All receive packets are in use and one transfer arrives. Does the USB thread wait? Does the device still answer a USB control request (GET_CONFIGURATION)?
- One transfer carries 6 datagrams, and only 4 receive packets are free. How many datagrams reach the application?
The same result on all six targets, little-endian and big-endian:
| Variant | Test 1: USB thread waits | Test 1: control request answered | Test 2: datagrams delivered with 4 free packets |
|---|---|---|---|
| A | yes | no | 4, then the USB thread waits |
| B | no | yes | 3 |
| C | no | yes | 4 |
Real boards: an application that stops reading its socket
A slow application is the simplest way to empty the receive packet pool. A UDP socket in Zephyr keeps every datagram that the application has not read yet, and each datagram holds one receive packet. Our test firmware does this in a fixed cycle:
- Hold, 10 s: the application does not read its UDP socket.
- Drain, 5 s: the application reads the socket continuously.
The PC sends 200 UDP datagrams per second to the board. During each run:
- A monitor thread on the board checks every millisecond whether the USB thread waits for a receive packet.
- The PC sends a USB control request (GET_STATUS) to the board every 100 ms, with a time limit of 500 ms. A device answers this request in its USB thread, independently of the network.
- The PC records its network interface counters and the kernel log (the TX watchdog).
- The board prints one heartbeat line per second. On the Teensy, the console is a USB serial port in the same USB device, so the heartbeat shows whether other USB functions continue. On the other boards the console does not use USB (a UART, and SEGGER RTT through the debug probe).
Each run lasts 50 s, so it has two or three complete hold phases. We repeated the runs; the results did not change.
Results on the real boards
| Board | Variant | USB thread waits during hold | USB control requests without answer | PC TX watchdog | Datagrams the socket could hold |
|---|---|---|---|---|---|
| Teensy 4.1 | A | 99.6 % | 55 of 220 | yes, every hold | 8 of 8, then the device stops |
| B | 0 % | - | no | 7 of 8 | |
| C | 0 % | 0 of 507 | no | 8 of 8 | |
| E (ECM) | 99.6 % | - | yes | 8 of 8, then the device stops | |
| F (ECM) | 0 % | - | no | 8 of 8 | |
| STM32F3 | A | 99.8 % | 52 of 236 | yes, every hold | 4 of 4, then the device stops |
| B | 0 % | - | no | 3 of 4 | |
| C | 0 % | 0 of 505 | no | 4 of 4 | |
| E (ECM) | 99.8 % | - | yes | 4 of 4, then the device stops | |
| F (ECM) | 0 % | - | no | 4 of 4 | |
| Raspberry Pi Pico | A | 99.6 % | 49 of 254 | yes, every hold | 8 of 8, then the device stops |
| B | 0 % | 0 of 503 | no | 7 of 8 | |
| C | 0 % | 0 of 504 | no | 8 of 8 | |
| E (ECM) | 99.6 % | 48 of 260 | yes | 8 of 8, then the device stops | |
| F (ECM) | 0 % | 0 of 504 | no | 8 of 8 |
"-": the run was made before the PC had permission to send the control requests. The remaining 0.2 to 0.4 % in the A rows is the time to fill the packet pool at the start of a hold phase. With variants A and E, the USB thread waited in every complete hold phase of every run.
Each plot shows how many datagrams per second the PC sends to the board (blue). Gray bands are the hold phases, from the board's clock. Orange lines are the PC's TX watchdog events. With variant A, the PC stops sending soon after a hold phase starts, because the board no longer accepts USB transfers. When the hold phase ends, the backlog goes through in a short burst.
What the debugger shows
We stopped the STM32F3 with a debugger 8.2 s into a hold phase, with variant A. The USB thread was in this call chain:
usbd_thread() usbd_core.c:222
cdc_ncm_acl_out_cb() usbd_cdc_ncm.c:574 (K_FOREVER)
net_pkt_rx_alloc_with_buffer() net_pkt.c:1882
k_mem_slab_alloc(&rx_pkts) mem_slab.c:250
z_swap() the thread waitsAll 4 receive packets were in use, and the monitor thread had counted 7431 waiting samples of 7450 in that hold phase.
The CPU does not stop; the USB device does
Our first description said "lock-up". The measurements make this more exact:
- The CPU continues. The monitor thread took about 9,090 samples in each 10-second hold phase, with the old code and with the fix: it was not delayed.
- The whole USB device stops: the network interface, the standard control requests, and other USB functions of the same device. On the Teensy, 23 to 34 of the heartbeat lines on the USB serial console came more than 1 s late, the worst 8.6 s late.
- The device recovers when the application frees a packet. An application that never reads its socket stops the USB device permanently.
- The wait does not prevent data loss. After 5 s, the Linux driver's TX watchdog cancels the transfers that are in progress: 3 to 17 transfers per hold phase. The board then reports a gap in the transfer sequence numbers.
Problems found, and their status
| # | Problem | Evidence | Status |
|---|---|---|---|
| 1 | CDC-NCM waits for a receive packet in the USB thread; the whole USB device stops. | 3 boards, 6 emulated CPUs | Fixed: #123515, merged 29 October 2026 |
| 2 | CDC-ECM has the same wait (K_FOREVER in cdc_ecm_acl_out_cb()). | 3 boards | One-line fix prepared and tested; separate pull request to come |
| 3 | After fix 1, CDC-NCM still uses a temporary packet for each transfer. It takes one receive packet from the pool and costs one more allocation for each transfer. | The application can use one packet less (7 of 8, 3 of 4); emulated test: 3 instead of 4 datagrams | Patch prepared (suggested by a Zephyr maintainer during the review of #123515): copy each datagram directly; 196 to 220 bytes less flash |
| 4 | Under packet shortage, the drivers log one error line for each dropped datagram, with no rate limit (about 5000 lines in 35 s at 200 datagrams/s). | On the STM32F3 this pushed lines of the application out of the console buffer | Open |
| 5 | The default stack size of the USB device and USB host threads (1024 bytes) is too small on SPARC. The same test needs 1132 to 1180 bytes there, and 516 to 664 bytes on the other CPUs. With the default size, memory was corrupted without a message. | QEMU LEON3, thread analyzer | Open; issue in preparation |
| 6 | One run on emulated Cortex-M3 received a corrupt transfer from the virtual USB host and failed a reference-count assertion. | 1 occurrence; 0 in 6 repeats | Open; needs a reproducer |
One more observation is not a defect: when the pool is empty, the board also drops the PC's ARP requests, so the PC holds back its traffic until the address resolves. Every driver that drops packets has this effect.
Cost of the changes
The fixes add no RAM. Flash (code size, text) of the test firmware:
| Board | A | B | C | E | F |
|---|---|---|---|---|---|
| Teensy 4.1 | 138,756 | 138,752 | 138,536 | 137,572 | 137,568 |
| STM32F3 | 91,520 | 91,512 | 91,324 | 90,396 | 90,392 |
| Raspberry Pi Pico | 92,008 | 92,000 | 91,804 | 90,712 | 90,704 |
The static RAM (data + bss) is the same for A, B and C. The gain of C is at run time: the application can use every receive packet.
Next
- Send patches 2 and 3 to the Zephyr project as separate pull requests.
- Measure the CPU time of the receive path in clock cycles per transfer, for A, B and C, on the three boards.
- Send malformed NCM transfers to the driver, in emulation with address and undefined-behaviour sanitizers, and on the real boards, to check the bounds checks of the driver.