Live Speech-to-Text on a $5 Microcontroller: How a 65.7M-Parameter Model Runs Offline on the ESP32-P4

A five-dollar microcontroller now writes down full English sentences as people speak — live, on the device itself, with no internet and no cloud. It is the most practical demonstration yet of ESP32-P4 offline speech recognition, and it is fully open source.

esp32-p4-offline-speech-recognition hero
Live Speech-to-Text on a $5 Microcontroller: How a 65.7M-Parameter Model Runs Offline on the ESP32-P4

Built by developer Boumedine Billal, the project is called esp32-pie-asr: plug a microphone in, start talking, and words appear on screen almost as fast as you say them. No audio is uploaded, no server asked, no subscription needed.

Privacy is the obvious win; the quieter one is accessibility — live captions for meetings, lectures, or conversation, for about the price of a cup of coffee. For deaf and hard-of-hearing users, a captioner that never needs a network is quietly revolutionary.

How ESP32-P4 offline speech recognition works

The ESP32-P4 is a microcontroller — a small, cheap, low-power computer on a chip made by Espressif. It has two cores on the open RISC-V design and 32 MB of PSRAM: auxiliary memory that gives the little chip room to breathe. What it lacks is an NPU — a neural processing unit, the dedicated circuit that speeds up AI on bigger hardware.

esp32-p4-offline-speech-recognition infographic

That matters: speech recognition on microcontrollers used to mean fixed commands — Espressif’s own toolkit, ESP-SR, recognizes a fixed menu of up to 200 spoken commands. Writing arbitrary sentences is a different league — it needs an ASR model (automatic speech recognition: software that turns audio into text) with tens of millions of internal parameters — the adjustable dials learned during training.

The Model That Would Not Fit

The heart is Kroko, a speech model from the company Banafo. It belongs to the Zipformer2 family of neural networks and is a transducer: an ASR model that produces words while you are still talking, instead of waiting for you to stop. That streaming behaviour is what makes live captions possible.

Kroko has 65.7 million parameters. As released, its weights — the number settings that hold its skill — take about 71 MB even in int8 form (8-bit integers, whole numbers with 256 possible values). The board’s 32 MB of PSRAM could not hold them, and standard toolkits were no help either, so Billal wrote his own inference engine in C and solved the size problem with quantization — storing numbers at coarser precision.

Squeezing It Down: From 71 MB to 26.65 MB

Instead of 8 bits for every weight, most get just 2 (four possible values) and a smaller share gets 4 (sixteen). Which matrix gets which is chosen by sensitivity: the layers that would lose the most accuracy keep 4 bits, the rest drop to 2 — with no retraining.

Two refinements make the coarse dials behave: a Hadamard rotation reshuffles the numbers so no extreme value overshoots its range, and rounding uses GPTQ, which nudges neighbouring values so small errors cancel instead of piling up. The result: 26.65 MB, with an accuracy loss that matters mainly to benchmark watchers.

Even 26.65 MB does not fit the board’s 16 MB of flash memory, so the weights live on a microSD card: at boot, the firmware (the software running the board) loads them into the 32 MB PSRAM in under three seconds.

Faster Than You Speak

Fitting is half the battle; speed is the other. A streaming model must process audio faster than it arrives, or the transcript falls behind and memory grows forever. The first port took 1.3 seconds of processing for every 1.28 seconds of audio — behind schedule.

The fix had two parts. First, the inner loops were hand-written in assembly for PIE, the chip’s vector math extension — its SIMD capability (single instruction, multiple data), letting one instruction do the same math on sixteen numbers at once. Second, every calculation runs on integers — whole numbers, not decimals — which these cores love. The engine handles each 1.28-second chunk in about 0.90 seconds: faster than real time, with constant memory however long the talk runs. A word becomes final, locked into the transcript, about two seconds after it was said.

The chip even computes exactly what the PC build does, bit for bit — the usual math libraries were replaced with deterministic versions — so accuracy measured on full test sets is the board’s own.

About Four Wrong Words in 100

Accuracy is graded with WER — word error rate, the share of words a recognizer gets wrong. On LibriSpeech test-clean, a widely used benchmark of careful read-aloud speech, this project lands around four wrong words in every hundred, close to what the same model scores on a PC.

Honest limits: the system is English only, and those numbers come from clean read speech — noisy rooms and strong accents remain hard. It needs the newest chip revision (ESP32-P4X), and power consumption is unmeasured.

Voice Control Without a Cloud

Transcription is only half the demo. The same firmware turns spoken orders into actions: six demos drive a simulated robot vacuum, smart home, car cabin, factory line, kitchen, and presenter — no language model, no internet. A text file on the microSD card lets users add custom words and phrases, steering what the recognizer listens for without retraining. In tests, command success was nearly perfect — zero false actions on near-miss clips.

Open Source and Reproducible

The code is Apache-2.0 on GitHub, the method documented in an arXiv paper, and the weights published on Hugging Face. Anyone with an ESP32-P4X board, a microphone, a screen, and a microSD card can flash the firmware — even from a browser — and the repository shows how to reproduce the accuracy numbers on a PC first.

It is the same pattern as running LLMs locally on a Jetson Orin Nano: big models, small hardware, data that never leaves the room. On the roadmap: an ESP32-S31 port and a future Arabic model. For five dollars, that is a lot of future.

For the technical reader

  • Chip: ESP32-P4X (chip revision v3), two RISC-V cores at 400 MHz, 32 MB PSRAM, 16 MB flash, no NPU; Espressif ESP32-P4X-Function-EV-Board (ES8311 codec, 1024×600 screen, microSD).
  • Model: Kroko (Banafo), 65.7M-parameter streaming Zipformer2 transducer, 650 BPE tokens (vocabulary pieces).
  • Weights: released int8 71 MB → 2/4-bit 26.65 MB; Hadamard-rotated groups of 128, one float16 norm per group; bit width per matrix from measured sensitivity; GPTQ rounding; no retraining.
  • Accuracy (LibriSpeech, beam search with 4 hypotheses): test-clean 4.34% WER, test-other 10.26%; released int8 model on PC: 3.94% / 9.17%. FLEURS English: 15.95%; Common Voice 17.0: 19.56%.
  • Speed: 0.70× real time — each 1.28 s chunk in ~0.90 s on both cores, screen included; word final ~2 s (median 1.9–2.3 s, 95% within 4.1 s).
  • Memory: weights and state in 32 MB PSRAM; ~121 KB internal RAM free (104 KB with demos); weights load from microSD at boot in ~2.9 s.

Reproducing the WER measurements:

git clone https://github.com/BoumedineBillal/esp32-pie-asr
cd esp32-pie-asr
python models/download.py --card <the microSD card's folder>
cmake -S pieasr/host -B build/host -DPIEASR_WEIGHTS=$PWD/models/kroko-en-q24-v1.bin
cmake --build build/host && ctest --test-dir build/host
python tools/eval/get_librispeech.py --set test-clean
bash tools/eval/run_wer.sh data/test-clean int+beam4

Hardware used in this guide

Links open search results on Amazon.in. As an Amazon Associate we earn from qualifying purchases.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top