A lock-up in the Zephyr USB network driver, and a two-line fix
Many small Zephyr boards have no Ethernet connector. They connect to a PC with Ethernet over the USB cable, through the USB CDC-NCM driver of Zephyr. Under load, this driver could stop the whole USB device: no network, no serial console, until a reset. We found the cause while we tested an encrypted network client on a Teensy 4.1. The fix changes two lines. We offer it to the Zephyr project: pull request #123515.
The symptom
- The board stopped all USB communication: the network and the USB serial console.
- The Linux PC logged
cdc_ncm ...: NETDEV WATCHDOG: transmit queue 0 timed outevery 5 seconds. - Only a reset of the board helped.
- It happened under load, for example a short flood of 800 UDP datagrams per second.
An open Zephyr issue, #102857, shows the same symptom on a Raspberry Pi Pico after an nmap scan.
How we found the cause
A stopped board cannot tell you what is wrong, because its console is also stopped. We added a small watchdog thread to our test firmware. Every second, it read how many free network packets were available. When no packets were free for 8 seconds, it wrote the state of all threads into a RAM area that a reset does not clear. Then it reset the board. After the reset, the firmware printed this record.
The record showed that all threads were waiting, also the USB thread. From there, we read the driver code.
The cause
Zephyr runs all USB work of a device in one thread. When the PC sends Ethernet data, this thread copies the data into network packets. The driver asked for these packets like this:
- For a temporary packet, it waited up to 100 ms. It also took this packet from the pool for transmitted data.
- For each received Ethernet frame, it waited with no time limit (
K_FOREVER).
Under load, the network stack had no free packets. So the USB thread waited. But the stack frees packets only after it sends its replies, and in our Zephyr version the send function waited for the same USB thread. Each part waited for the other, so the device stopped until a reset.
The fix
The receive function must never wait. If there is no free packet, it now drops the received data and continues. A normal Ethernet driver does the same when its receive buffers are full. The temporary packet now comes from the pool for received data, which is the correct pool.
- src = net_pkt_alloc(K_MSEC(NET_PKT_ALLOC_TIMEOUT));
+ src = net_pkt_rx_alloc(K_NO_WAIT);
...
- pkt = net_pkt_rx_alloc_with_buffer(data->iface, len, NET_AF_UNSPEC, 0, K_FOREVER);
+ pkt = net_pkt_rx_alloc_with_buffer(data->iface, len, NET_AF_UNSPEC, 0, K_NO_WAIT);
The change also removes one constant that is no longer used. Under a flood, the board now drops packets and logs a message for each drop. It does not stop.
Test result
- Before the fix: 1 of 4 test runs in the last batch stopped the board permanently, and several earlier runs too.
- After the fix: 6 of 6 runs passed. Each run started the board, sent 800 datagrams per second for 10 seconds, and then sent 400 test messages at a low rate. All 400 test messages came back.
- The board is a Teensy 4.1 with Zephyr v4.4.2.
Notes for other users
- The current Zephyr main branch already limits the wait in the send function. On main, the same problem should cause a long stall instead of a permanent stop. The receive function still waits, so the fix is still necessary. We have not tested on main yet.
- The USB CDC-ECM driver has the same wait in its receive function. We plan a second, separate fix.
- Until the fix is in a release, you can apply the two-line change to your own Zephyr tree.
Next
This is our first change to the core of Zephyr. We thank the maintainers of the USB device stack for code that was easy to read and to test. We will follow the review of the pull request and report the result here.