ZeroTier on a microcontroller: an encrypted network leaf for Zephyr
ZeroTier creates a private, encrypted Ethernet network over the internet or a local network, and it normally runs on PCs, servers and routers. We got it running under Zephyr on a Teensy 4.1 (Cortex-M7, 600 MHz, 1 MB RAM). The board joins a private ZeroTier network alongside stock ZeroTier 1.16.2 peers and a stock controller, with encryption ending on the board itself rather than on a gateway. Ordinary Zephyr services like HTTP, Telnet and SNMP can then use the encrypted network without changes. The code is open source (MPL-2.0) at github.com/jerzydziewierz/zephyr-zerotier-leaf.
Why this is not a simple port
Although the ZeroTier One core is written in portable C++, it assumes PC-scale memory. Two allocations in the upstream source take more than 4 MiB by themselves:
- Identity validation uses a memory-hard hash with a 2 MiB work area, which the core runs for its own identity and for every new peer.
- The receive queue pre-allocates 32 entries, each sized for a full packet plus six full fragments, totaling about 2.2 MiB.
On the board, the normal application RAM region is 256 KiB. Simply lowering the MTU or talking to fewer peers does not get rid of those two allocations.
The approach: a closed-fleet leaf
Instead of trying to join arbitrary ZeroTier networks, the board acts as a restricted leaf in a small network that you control:
- Identity check moves to provisioning. A trusted host validates each full public identity ahead of time with the upstream tool (
zerotier-idtool validate). The board accepts only those exact identities, at most seven (roots, a controller and up to three management machines), and rejects any other node before allocating memory for it. - All other security checks stay. Key agreement, packet authentication, signed network configuration, membership certificates and rules all run unchanged upstream code, and the controller still has to authorize the board.
- Every growing resource has a limit. We cap the core at one network, one IPv4 address, four receive and four transmit slots, packets up to 2048 bytes and two fragments, plus fixed limits for rules, routes and certificates. Configurations that exceed these bounds are rejected outright, never cut short.
- The core has its own heap. The build redirects the core's
newanddeleteto a private 64 KiB heap, and a build check fails if any reference to the global allocator remains. Even if a hostile peer fills that heap, the rest of the firmware keeps running.
The ZeroTier changes live in four reviewed patches against the pinned 1.16.2 source, verified at build time so the firmware cannot accidentally build against an unreviewed vendor tree. On the Zephyr side, the module provides a native Ethernet interface, a UDP transport thread, and start, status and stop functions.
Results
- RAM: the module adds 110,508 bytes to the linked image against a 128 KiB budget. On the board, the core heap peaks at 43 of 64 KiB and the worker stack at 20.7 of 26 KiB.
- Interoperability: host tests pair the restricted core against an unmodified reference core. Normal mode delivers frames and ARP as expected, while denied-membership and tampered-packet modes deliver nothing.
- Hostile input: a host test corpus exercises forged MACs, certificates and configurations, along with 288 authenticated malformed inputs and 2,048 raw and fragment inputs.
- Soak: an application serving HTTP, Telnet and SNMP over the overlay ran for 4.15 hours across 10 cold boots by reflash. HTTP over the overlay answered 160 of 160 requests and SNMP 32 of 32, and the private heap did not grow.
- Throughput and loss: with the latest buffer setting (see below), the board lost 0 of 12,000 echoes of 1200 bytes at 50 per second. The rate sweep in the plot below ran before that change: it reached about 200 echoes per second with less than 1.5% loss and about 7 ms round-trip time. Software AES on the CPU is the bottleneck: one decrypt and one encrypt cost about 2.4 ms.
One finding: signature checks and receive buffers
During the soak test, we saw about 2 lost echoes per minute, always at the same times. The module's diagnostic command (zt diag) pointed to the cause: every 60 seconds the core blocks to verify Ed25519 signatures, spending about 47 ms on the controller's configuration refresh and about 94 ms on a peer's membership certificate. While those checks run, the transport thread is not reading the network. Zephyr's 36 default receive buffers only hold about 3 encrypted datagrams, so the USB driver dropped whatever arrived next.
The fix was configuration rather than code: setting CONFIG_NET_BUF_RX_COUNT=96 at a cost of 10 KiB more RAM. With the larger buffer pool, 0 of 12,000 echoes were lost. We do not skip or delay the signature checks, since doing so would let the board accept membership or configuration that it had not verified.
On-board testing also uncovered a lock-up in the Zephyr USB network driver, which we covered in our previous post and sent upstream in Zephyr pull request #123515.
Limits
This is a prototype and is not ready for production. So far we have only tested it on one board type (Teensy 4.1, Zephyr v4.4.2) using USB as the underlying link; we have not yet run on-board tests with seven live peers, hostile traffic, membership revocation and expiry, or physical Ethernet. Persistent key storage and authenticated roster updates are also unimplemented, so today the firmware image contains the board's secret key. Finally, ZeroTier 1.x has no forward secrecy, and the transport does not stop replay of an authenticated frame. The repository README and VALIDATION.md list all limits alongside the test evidence.
Next
Next up, we plan to test seven live peers and hostile input directly on the board, and measure where in the i.MX RT1062 memory the crypto code runs fastest. This project is not affiliated with or endorsed by ZeroTier, Inc., though we thank the ZeroTier authors for a core clean enough to restrict this way.