Teensy 4.1 and Zephyr USB networking: ten times faster from the same chip: report
This report covers the experiments behind the short article. The question: how fast can the Zephyr USB network stack go on a Teensy 4.1, and what limits it? The method is the one from the Pico report, adapted to a board without a debug port.
Set-up and metrics
Board: Teensy 4.1 (NXP i.MX RT1062, Cortex-M7 at 600 MHz, 32 KB instruction cache with 32-byte lines, 128 KB ITCM, 128 KB DTCM, code in an 8 MB QSPI flash read at 100 MHz). USB high speed, CDC-ECM (and CDC-NCM where stated), Zephyr main of 31 October 2026, a small UDP echo application, -Os unless stated. All images use the same source code; only placement and configuration change. Zephyr places nothing in ITCM by default; all data is in the on-chip OCRAM.
| Metric | How it is measured |
|---|---|
| Peak UDP echo rate | The host sends 32-byte UDP datagrams at fixed rates (2 s steps, 500/s apart up to 30,000/s) and counts the replies; 3 passes, each after a new flash of the image. Peak = highest reply rate of a pass; mean ± SD over passes. |
| Lossless rate | The highest offered rate up to which every step of every pass was answered at ≥ 99 %. |
| Turnaround | usbmon on the host: end of the OUT transfer to end of the IN reply, median of 300 pings. |
| Profile | On-target PC sampler (below): share of busy time per memory, library and function. |
Is it the memory? Three images, no profiler
| Image | Turnaround | Peak UDP echo |
|---|---|---|
| Stock | 0.183–0.198 ms | 2,257 ± 8/s |
| Network + USB libraries in ITCM (48 KB) | 0.062–0.065 ms | 12,065/s (1 pass) |
| Instruction cache off | 2.945 ms | 273/s (1 pass) |
| Data cache off | 0.816 ms | 693/s (1 pass) |
An uncached line fill from this QSPI flash costs about 500 core cycles. The cache hides most of them, but not enough.
An on-target PC sampler
The Teensy has no SWD or JTAG connector. A board with the same chip and a debug port would have a different flash chip and flash set-up, which is the thing under study, so I did not use one. Instead, a periodic interrupt timer channel interrupts at Zephyr’s zero-latency priority (so the kernel’s interrupt lock does not delay it) and stores the program counter from the exception frame. The handler is in ITCM and the 16,384-entry buffer in DTCM, so the sampler uses neither the caches nor the flash. The interval varies by ±25 % to avoid aliasing with periodic work. With the sampler built in but idle, the turnaround did not change (0.043 vs 0.042 ms). A second mode also stores the link register, which shows the caller of a short function.
Stock image at 1,500 echoes/s: 76 % of the time busy (about 300,000 cycles per echo), all of it in flash, in 347 functions that execute 1,013 distinct 32-byte lines: 32 KB, the full size of the instruction cache.
Placement
| Image | ITCM | Peak UDP echo |
|---|---|---|
| Densest functions by name, 8 / 16 / 32 KB list | 9 / 17 / 34 KB | 4,139 / 9,548 / 15,921/s |
| Network + USB libraries | 48 KB | 12,065/s (1 pass) |
| + kernel, context switch, timer, NXP EHCI controller driver | 64 KB | 15,919 ± 5/s |
| + MPU, events, message queues, poll (rest found in flash by the profiler) | 66 KB | 15,932 ± 3/s |
The 32 KB name list, chosen from the stock profile by samples per byte, equals the 66 KB of whole files. Moving the last flash-resident hot code (15 % of busy samples) changed nothing: that code was already cache-resident. From here the echo path is compute-bound: about 39,000 cycles per echo.
Code that runs before Zephyr copies code to ITCM must stay in flash (the SoC’s SystemInit, memcpy/memset used by the copy, the reset path), or the image does not boot.
Tuning after placement
| Change | Peak UDP echo |
|---|---|
-O2 | 17,785 ± 8/s |
-O2 + all data in DTCM | 18,379 ± 103/s |
-O2, no time slicing | 18,839 ± 1/s |
-O2, no MPU stack guard | 19,003 ± 8/s |
-O2 + DTCM + no stack guard + no time slicing | 20,807 ± 1/s |
| The same with CDC-NCM | 22,359 ± 32/s |
No network RX/TX threads (± -O2) | 10,464 / 11,000/s, collapses to about 1,000/s under overload |
| CDC-NCM, stock (for comparison) | 2,658 ± 103/s, collapses to about 450/s under overload |
On the Pico, -O2 made things worse because larger code missed the cache more. Here, with the code in ITCM, it helps. No stack guard and no time slicing each remove work from every context switch (MPU reprogramming, a slice timeout); the stack guard is a safety feature, so that gain has a price. The 32 KB name list made from the -Os profile still works on the -O2 build (17,785/s with 39 KB of ITCM): the names survive a change of compiler options.
What is left
Best NCM image at 10,000 echoes/s: about 27,000 cycles per echo. Kernel 36 %, IP stack 30 %, NXP USB controller driver and memcpy 15 %. The largest single item is the kernel mutex (17 % of busy time). A micro-benchmark gives 150 cycles per uncontended lock and unlock, so about 30 pairs per echo. The link-register samples show who takes them: mostly read-only IPv4 address checks (broadcast check, address lookup, best-match, on-link, route lookup) that lock the interface. With the interface lock removed (unsafe, measurement only) the peak is 24,838/s, +11 %.
Larger datagrams
With 1000-byte requests (2 passes; the test application answered with the first 256 bytes, so this is 1000 B in, 256 B out): CDC-ECM stock 1,926/s, best 14,422/s (14.4 MB/s towards the board); CDC-NCM stock 2,002/s, best 11,844/s. With large datagrams ECM is ahead of NCM, with small ones NCM is ahead; I have not analysed why.
Reproduction with the published sample
The test application above also contains instrumentation for other work. The repository zephyr-codeplace has a minimal sample (samples/usb_udp_echo: CDC-NCM, blocking socket loop, 1500-byte buffer) with the same placement as build options. Measured with it on the same board:
| Build | Fast memory | Peak UDP echo |
|---|---|---|
| Stock | 0 | 2,238/s |
| 32 KB function list | 32 KB ITCM | 16,246/s |
| Hot path (libraries + kernel and driver files) | 66 KB ITCM | 16,943/s |
+ -O2, no stack guard, no time slicing | 77 KB ITCM | 21,370/s |
| + data in DTCM | 77 KB ITCM | 22,586/s |
With 1000-byte datagrams echoed in full: 1,678/s stock, 10,812/s with the fastest build (10.8 MB/s in each direction). One difference to the test application: with the sample's simple blocking loop, the fast builds lose some replies between about 6,500 and 14,000 requests per second and then echo everything again at higher rates. I have not analysed this yet.
Limits and open points
- One board, one Zephyr version, one echo application; ITCM is free here, a real application shares it.
- The turnaround of the fast images varies between runs (for example 0.04 and 0.08 ms for the same image); throughput is stable to within 1 %.
- Zephyr finding on the way: with the data-TCM node as the system RAM, data placed with Zephyr’s DTCM section macro overlaps the normal RAM contents without any link error.
- The overload collapse of stock CDC-NCM is not analysed.
Evidence: Teensy 4.1, Zephyr main of 31 October 2026, 1 November 2026; peaks are means of 3 independent passes (each after a new flash) unless stated.