An agent that lives on the sensor.
A self-contained sensing node — camera, microphone, and on-device intelligence — that decides what it's looking at on its own, then pings you wherever you already chat. No cloud round-trip. Tested on the bench, working. Here's how each piece is wired, and why each piece is there.
on-device AI · WiFi + camera + mic sensing · an agent on the chip · alerts to any chat terminal
What if the edge device decides what its sensors mean — before anything leaves the chip?
An agent on the cloud can read a JPEG. An agent on a chip can't. The whole post is about how to close that gap without putting the chip on a leash.
The standard cloud setup is uncomplicated: a camera takes a picture, ships it to a vision model, gets back text, and the text feeds a tool-calling LLM. The LLM decides. It works — until you measure it.
- Latency: seconds per round-trip. Unacceptable for "is there a person in the room right now?"
- Bandwidth: a 200 KB JPEG every few seconds across the WAN. Pile up a dozen sites and you're streaming.
- Failure mode: the second the WAN flaps, the agent is blind. Not degraded — blind.
- Privacy: raw pixels leave the building. For some workloads that's a non-starter.
The fix is to move the "what does this picture mean?" step onto the chip. Run a quantised neural net locally, get a short text label out, and let the agent reason over the label. Cloud is then an escalation, not a dependency.
The agent never touches raw pixels or audio. TinyML converts media into a short text label. The label is the tool result.
One chip does all of it.
The hardware is a Seeed XIAO ESP32-S3 Sense. Dual-core ESP32-S3 at 240 MHz, 8 MB of PSRAM, 8 MB of flash, native USB-C, an OV-series camera on the DVP/LCD_CAM peripheral (the unit on our bench carries an OV3660; the kit also ships with OV2640), a PDM MEMS microphone on I2S, a microSD slot. About the size of a thumbnail. About $14 retail. The chip itself is roughly $4 in volume.
S3 specifically — not C3 or C6 — because two things need silicon support: the camera needs the DVP/LCD_CAM interface and the mic needs I2S-PDM-RX. C-series chips don't have either. The S3 also has SIMD instructions that ESP-NN uses to make TFLM run at usable speed. Without ESP-NN, inference is slow enough to be a deal-breaker.
(The detection model doesn't care which OV-series sensor is fitted — it consumes a downsampled 96×96 grayscale buffer either way.)
The OV2640 captures, the PDM mic captures, neither of them think.
The camera grabs at whatever resolution the next stage needs. For person detection, that's a downsampled 96 × 96 grayscale buffer — about 9 KB per frame, not the 600 KB of a full JPEG. For escalation captures (the ones that fly to Telegram), it's a higher-res RGB565 buffer that gets software-encoded to JPEG.
The mic captures 16-bit PCM at 16 kHz over I2S-PDM. The buffer used for level sensing is 200 ms long (3200 samples). Short enough that a loud transient doesn't get smeared.
Both buffers live in PSRAM. Both are short-lived — they're consumed by the next stage and freed before the agent loop wakes up. The chip's internal SRAM (512 KB) stays available for WiFi/TLS handshakes and the zclaw heap.
The camera isn't passive, either. The focused main sensor and its flash LED are both on GPIO the agent can drive. So the capture step isn't "grab whatever's there" — it's: is the scene dark? turn the flash on. is the subject close? nudge focus. then grab. A dark corridor that would give a camera-first system a black frame gives EspCamML a lit, in-focus 96×96 — because the agent lit it on purpose, a few milliseconds before the shutter. The same GPIO that fires the flash for a Telegram photo also gives the detection model a usable frame in low light. One tool call: led on → capture → led off.
Pictures become words. Sounds become words. Inside 100 milliseconds.
TFLM (TensorFlow Lite Micro) is Google's MCU inference runtime, ported to ESP-IDF by Espressif as the esp-tflite-micro component. It ships two reference models that we use as starting points:
| Model | Input | Model | Tensor arena | What it does |
|---|---|---|---|---|
person_detection | 96×96 grayscale | ~250 KB | ~100–200 KB | binary "person / no person" |
micro_speech | ~1s audio → log-mel | ~20–40 KB | ~46 KB | keyword spotting |
Both models are int8-quantised — that's why they fit on a microcontroller at all. Inference is matrix math in int8, ESP-NN accelerates the heavy kernels with the S3's SIMD instructions, and the runtime hands back an array of int8 logits that we dequantise to probabilities.
The whole pipeline is pure label math: argmax over the int8 scores, dequantise the winner, format as "person 0.92". Eight bytes of text instead of nine kilobytes of pixels. That's the entire trick.
The label fits the 512-byte tool-result contract zclaw uses natively. It also fits inside an LLM prompt, a database row, a Telegram message — anywhere a string fits. That portability is the point of compressing the modality through inference instead of shipping the modality itself.
The agent only reads text. So we only feed it text.
zclaw is a small, opinionated tool-calling LLM runtime that ships on the ESP32. It has a text-only loop: the model emits a tool call, the runtime executes the handler, the handler returns a ≤ 512-byte string, the model reads the string and decides what to do next.
Most of the tools you'd expect are already there — gpio_set, cron_add, memory_get, memory_set, http_get. EspCamML adds a thin family of new tools on top, each of which speaks the same string-in / string-out contract:
| Tool | Returns | What it does on-chip |
|---|---|---|
camera_capture | "jpeg ok 38 KB" or error | grab one JPEG, hold it for an escalation tool |
camera_motion_check | "motion 0.31" | frame-to-frame diff over a small grid; no model |
camera_classify | "person 0.92" | TFLM person_detection on a 96×96 grab |
sound_level | "dbfs -34.7" | RMS of a 200 ms PCM buffer |
sound_tone | "tone 4kHz present" | Goertzel filter on a target frequency |
sound_classify | "glass_break 0.81" | TFLM audio model on log-mel features |
monitor_person | "started · 5s interval" | fire a FreeRTOS task that runs camera_classify on a loop and pings Telegram when it crosses threshold |
monitor_sound | "started · 1s interval" | same idea, for the mic |
camera_describe | "a man holding a phone" | OPTIONAL — base64+POST the JPEG to a vision endpoint |
camera_send_telegram | "sent · 38 KB" | JPEG → multipart POST → Telegram sendPhoto |
Three things to notice:
- The two
monitor_*tools don't run the LLM in the loop. The agent calls them once to start a background FreeRTOS task; the task then runs the TFLM check on its own clock and fires Telegram directly when something crosses threshold. The agent is freed up. The chip isn't waking the LLM every five seconds just to ask "anything yet?" - Cooldowns are built in. A person-detection hit at t=0 doesn't fire again until at least
cooldown_mslater. This is the difference between an alerting system and a Telegram-spamming machine. - Cloud is opt-in, never default.
camera_describeexists for the cases where "person 0.92" isn't enough and the operator actually wants a sentence. The agent decides when to escalate; nothing is uploaded by reflex.
It lands wherever you already chat. The terminal is swappable.
The agent needs exactly one thing from the outside world: a way to push a line of text and a photo to a human. Anything with a bot or webhook API does that job — Slack, WhatsApp, Signal, SMS, email, a webhook into your own ops tool. The node doesn't care; it just needs one outbound channel.
For the bench build we used Telegram, because it works without a custom app, without a notification service, and without anyone signing in to a portal — and the Bot API is two endpoints. But that's an implementation choice, not the point. Swap the two calls below for your terminal of choice and nothing else changes:
https://api.telegram.org/bot<TOKEN>/sendMessage— POST a JSON body withchat_idandtext. Done.https://api.telegram.org/bot<TOKEN>/sendPhoto— multipart POST: chat_id field, photo field with the JPEG bytes. Done.
The token and chat ID live in NVS under the espcaml namespace, alongside the WiFi creds. Provisioning is a one-time idf.py monitor session with three CLI commands. Once it's in NVS, the bot is configured for the lifetime of the device.
This is the bit the user actually sees. The chip wakes, sees a person, fires the photo. Two seconds later the operator's phone vibrates with a JPEG and a one-line caption. That's the whole product as far as the end user is concerned. Everything else is plumbing.
And it's not one-way. The bot is also the control surface — you talk back to it in plain language and the agent reconfigures itself. Here's a real session on the bench: sound alerts coming in by dBFS, then a single message — "make it more sensitive" — and the agent drops its own threshold and tells you what it did.
That screenshot is a real session on the bench. Sound alerts stream in by dBFS. Then one message — "make it more sensitive" — and the agent drops its own threshold from −13 to −30 dBFS, keeps the photo-on-alert behaviour and the 30-second cooldown, and tells you exactly what it changed. No app, no dashboard, no settings screen. The chat is the configuration surface, and the thing on the other end is a $14 chip on a desk.
Four properties stack.
- It works offline. Cut the WAN. The TFLM model still classifies. The agent still runs. GPIO still toggles. Only the Telegram leg degrades — and even that queues locally and flushes when WiFi comes back.
- It's fast. 96 × 96 person detection on the S3 with ESP-NN is around 200 ms. End-to-end (capture → classify → decide → POST) is well under a second.
- It's cheap. One board, no recurring cloud bill per inference. A Telegram bot is free.
- It's auditable. Every tool result is a string. Every decision the agent makes is the model emitting JSON with a tool name. The whole event log fits in a text file. There's no "the model just decided" — there's a sequence of
tool_call → tool_resultpairs.
The honest boundary.
This isn't a security system. It's not graded for life-safety. The two reference models (person_detection, micro_speech) are alarm-grade, not forensic-grade — they tell you something happened, not who and not exactly what. For richer answers we escalate to a vision API (camera_describe) — and that requires a working WAN and a paid endpoint.
Custom classes — "glass break", "child crying", "machine bearing failure" — need their own model. That model gets trained off-device (Edge Impulse is the obvious tooling), exported as an int8 .tflite, dropped into the firmware's models partition, and the agent grows a new tool. The training loop is its own post.
This isn't a replacement for cameras in every situation. It's a replacement for cameras in the situations where cameras are an expensive default — empty rooms, unmanned sites, low-light corridors, places where the question is "is there anyone there?" not "what does their face look like?"
Bench notes, plainly.
- Person detection on a 96 × 96 grayscale grab — ~200 ms inference on the S3 with ESP-NN. Confidence stable above 0.85 for clear in-frame subjects.
- Telegram
sendPhotofrom device — JPEG arrives on the operator's phone in 1–2 seconds end-to-end. Multipart framing handles the boundary correctly. - Background
monitor_personwith a 5-second interval and 60-second cooldown — fired exactly once per discrete event. No spam. - Camera capture during inference — both share the OV2640 cleanly because the camera FB lives in PSRAM and we pipeline rather than concurrent-access.
- Pure label math (argmax / dequant / Goertzel / motion-grid) host-tested with
gcc; 30 / 30 unit tests pass before anything sees the chip. - Cam2 (the companion AI-Thinker ESP32-CAM) serves
/captureand a live MJPEG/streamas a second eye — the S3 fetches stills fromhttp://cam2.local/capturevia the standardhttp_gettool.
EspCamML is the camera-side cousin of LatentField.
LatentField is the same idea applied to RF: read WiFi CSI, compress to 8 latent fields per frame, let an agent reason over the fields. EspCamML applies the trick to the visual modality: capture pixels, compress to a text label per frame, let an agent reason over the label. Same agent. Same contract. Different sensor.
CameraPlus stitches the two together: RF runs continuously, the camera wakes only on a CSI trigger, and the JPEG flies to Telegram via the EspCamML path. The 91.5% energy-reduction figure on CameraPlus is exactly the duty-cycle gap between the always-on radio and the on-demand camera.
Three layers, one operating model:
- LatentField — RF only. The chip's quietest mode.
- EspCamML — RF + on-trigger camera, both classified locally. Telegram for the human.
- CameraPlus — the product that wraps both into a deployable, monitored system. See /cameraplus/.
Where to look.
The firmware lives in two folders under EspCamML/:
firmware/— the S3 brain. Overlays the zclaw agent with five new files:tools_camera.c,tools_audio.c,tools_ml.c,tools_monitor.c,tools_cloud.c. About 1,200 lines added.firmware-cam2/— the AI-Thinker companion. Plain HTTP server:/capture,/stream,/led?on=…. About 200 lines. Dumb on purpose.firmware-display/— the punk-OLED status panel that's stamped through the rest of this site as its visual signature. Pixel-art glyphs, torn paper, ink bleed. (See LatentField for the gallery of frames it cycles through.)
The host-testable bits compile under plain gcc with -DESPCAMML_HOST_TEST. The device-only paths (TFLM, I2S, esp-camera) need the IDF toolchain.
Three things to get started.
- A XIAO ESP32-S3 Sense. About $14. Ships with the camera and mic.
- A Telegram bot. Talk to
@BotFatherto make one. Free. You'll get a token and a chat id. - The ESP-IDF toolchain.
idf.py flash monitoris the whole workflow.
Want to talk about a deployment, or want to see it on a bench in person? Drop us a line — ai.operations@radioqubits.com.