Memory bandwidth, not TOPS, decides what runs at the edge
Teams compare TOPS, but tokens per second follow memory. A 30B model runs on a 16 GB Pi; a microcontroller "LLM" remains a demo. Ship the smallest tier that passes.
In brief
- In Jeff Geerling's January 2026 tests, the Raspberry Pi 5's own CPU beat a 40-TOPS Hailo-10H accelerator on nearly every language model.
- Token generation is memory-bound, so memory capacity and bandwidth set the ceiling that a compute rating ignores.
- Pick the lowest tier that clears your evaluation at production context and thermal limits, with flash reserved for rollback-safe updates.
A Raspberry Pi 5 with 16 GB of memory generated 8.03 tokens per second from Qwen3-30B-A3B in a run reported on Hacker News in January 2026. Embedded teams now ask whether a Pi, a Jetson or a neural processing unit (NPU) add-on can run a useful model, and whether a microcontroller (MCU) language model is more than a demo.
We draw three conclusions. Single-board computers (SBCs) can now run capable mixture-of-experts models for narrow, short-context jobs. MCU "LLMs" remain demonstrations; the MCU's real job is still keyword spotting, anomaly detection and classification.
The deciding numbers are memory capacity, memory bandwidth and tokens per second at production context length; TOPS (trillions of operations per second) ratings rarely predict any of them. And before choosing hardware, check whether a classifier would do the job.
Three tiers cover edge inference, and only two of them generate text
Match the tier to the job, and climb only when the tier below fails your evaluation.
| Tier | Hardware | Good for | Not for |
|---|---|---|---|
| 1 | MCU (ESP32-S3 class) | Wake words, command sets, anomaly flags | Any generative output |
| 2 | Pi 5 or Jetson, sub-1B to 4B models | Routing, extraction, short summaries | Long context, open-ended chat |
| 3 | 16 GB SBC or Jetson, MoE models up to ~30B | On-premises assistants with cloud fallback | Multi-step agents, long documents |
Source: Wennovate framework, drawing on the benchmarks cited in this article.
In our own builds, narrowing the task to structured extraction is usually what lets the second tier suffice.
Mixture-of-experts models put 30B-class capacity on a 16 GB board
The January run used a 2.70 bits-per-weight quantisation, which stores each of the model's weights in fewer bits. Jeff Geerling reproduced it in the thread, with caveats.
The model segfaulted (crashed) at default settings until he capped context, the text the model holds in view, at 4,096 tokens. The board also needed active cooling.
Generation then ran at about 6 to 7 tokens per second, falling to 4 to 6 on longer outputs.
In a mixture-of-experts (MoE) model, only a fraction of the weights is active for each token; the "A3B" in the name means about three billion. All 30 billion must still fit in RAM, but each token reads only the active slice. Per-token cost looks like a small dense model's; capacity looks like a large one's.
One commenter in the thread warned that short contexts produce coherent-looking answers that degrade on longer tasks. Others running them with Home Assistant, the open-source home-automation platform, said voice hardware, wake words and stale conversation state were bigger problems than the model. Small local models handled lights and fans; harder questions still went to the cloud.
Memory caps generation speed, and compute caps the wait for the first token
Token generation on these boards is usually memory-bound. Each token must stream the active weights through the memory bus, so tokens per second is capped at roughly bandwidth divided by bytes read per token. Compute ratings say little about it.
Prompt processing is the other half, and it is compute-bound. With the 30B model, Geerling measured 8 to 10 tokens per second of prompt processing on the Pi.
At that rate, a full 4,096-token context takes around seven minutes before the first output token. Short prompts hide this; retrieval and conversation history expose it. Time to first token at the real prompt length is the number to put in the requirements.
A 40-TOPS accelerator lost to the host CPU on language models
The Raspberry Pi AI HAT+ 2 shows how little a TOPS rating predicts. It pairs a Hailo-10H NPU, rated at 40 INT8 (8-bit integer) TOPS, with 8 GB of on-board LPDDR4X memory.
In Geerling's January 2026 tests, the Pi 5's own CPU beat the US$130 board on nearly every language model. Only Qwen2.5-Coder 1.5B came close.
Vision was different. A YOLO object-detection model ran about ten times faster than on the CPU, at around 30 frames per second. Mixed vision-and-LLM mode crashed. His verdict was that the software lags the hardware.
Power is the HAT's best argument. It is limited to 3 W, while the Pi's system-on-chip (SoC) can draw about 10 W under load. On a solar-powered enclosure, that difference can decide the design.
The Jetson Orin Nano Super sits at the other end of the trade. It lists 67 sparse INT8 TOPS and costs US$249.
Its memory bandwidth is 102 GB/s, up from 68 GB/s before the "Super" update, which makes it fast on models that fit.
Capacity is the catch. Its 8 GB of LPDDR5 memory cannot hold the 30B MoE model above. A 16 GB Pi fits bigger models and runs them slowly; the Jetson runs smaller models quickly. Choose by the model you need, then confirm with a benchmark at your real context length.
Microcontroller language models fail on updates before they fail on speed
A July 2026 write-up of a 28.9-million-parameter model with 4-bit weights, running on an ESP32-S3, shows where it stands.
With weights spread across flash, PSRAM (external pseudo-static RAM) and on-chip SRAM, it reaches about 9.7 tokens per second. Memory bandwidth again sets the pace: the output head alone takes 56% of each token's time.
The model cannot follow instructions or reliably emit JSON. The project's own documentation rules out question answering.
Two details matter more than speed. The model leaves about 4.4% of its flash partition free on a 16 MB part. That leaves no room for an A/B partition pair, the two firmware slots that let a failed over-the-air (OTA) update roll back.
And Espressif's speech-recognition framework runs on the same board, supporting up to 200 command words.
Its MultiNet7 model averages about 11 ms per frame, using roughly 2.9 MB of PSRAM. For a device that must respond to "open valve two", the dedicated model wins on fit, latency and update safety.
A prompted model tests the feature; a trained classifier usually ships it
In a 2025 Hacker News thread, one commenter asked whether a dedicated time-series architecture would make more sense than an LLM. Salvatore Sanfilippo (antirez) answered that a specialised network demands data collection and training; a downloaded model and a prompt does not. He described small models as a "bag of algorithms" for simple tasks.
We think both are right at different stages. A prompted small model is the fastest way to test whether a feature is worth building. A trained classifier, keyword model or rules engine is usually cheaper to validate, smaller, more predictable and easier to update across a fleet. Prototype with the prompt; ship the smallest model that passes your evaluation.
Rural coverage gaps make the Western Canadian case for local inference
Innovation, Science and Economic Development Canada (ISED) tracks progress toward universal 50/10 Mbps access.
In 2024, that service reached 87.4% of rural households, against 99.8% of urban ones.
Those are household numbers. Orchards, rangeland, forestry blocks and mine sites in the BC Interior and Alberta often sit well outside them. A device there must decide locally and sync when the link returns, with cloud inference as an upgrade path, not a dependency.
Put it to work: Three tests before you commit to edge hardware
- Measure sustained tokens per second and time to first token at your production prompt length, inside the enclosure at its thermal limit. Which tier passes?
- Benchmark any NPU add-on against the host CPU on the model you will ship.
- Reserve flash for an A/B update pair before budgeting it for weights, and try a trained classifier or keyword model on the task first.
The smallest tier that passes offline is the right one
The costliest edge mistakes start with the spec sheet. Choosing hardware by TOPS buys compute that a memory-bound workload cannot use. Quoting tokens per second at a context the product will never ship hides the wait users will face.
Two more surface after launch. A microcontroller LLM that fills its flash leaves no room for rollback-safe updates, and a device that depends on the cloud fails wherever it outlives the signal.
The useful question is not whether a model can run on the board. Find the smallest tier that passes your evaluation, offline, at the context your users will create.
Next in this series: The EU's 24-hour reporting clock already covers shipped devices
Building connected or robotic products? Book a 30-minute discovery call with the Wennovate team. No pitch deck required. Start with a scoping call
Sources
- A 30B Qwen model walks into a Raspberry Pi and runs in real time, Hacker News, January 2026.
- Raspberry Pi's new AI HAT adds 8GB of RAM for local LLMs, Jeff Geerling, January 2026.
- $249 NVIDIA Jetson Orin Nano Super Developer Kit targets generative AI applications at the edge, CNX Software, December 2024.
- Running a 28.9M-Parameter LLM on an $8 Microcontroller - Where ESP32-S3 Inference Actually Breaks, braindetox.kr, July 2026.
- What even is a small language model now?, Hacker News, May 2025.
- Progress toward universal access to high-speed Internet, Innovation, Science and Economic Development Canada, May 2026.
