A TinyStories model on my RP2040 board
Training, quantizing, and physically verifying a tiny Transformer on a custom Raspberry Pi Pico display board through intention-and-observation collaboration with an AI agent.
From training to the board
- Apple M1 MacTraining
- INT8 weightsportable C implementation
- RP2040 itselfLocal story generation
Recorded static SRAM
143,504 / 270,336 bytes · 53.1%
Budget less recorded static SRAM: 126,832 bytes. This difference is not measured runtime headroom.
- Model weights
- 98,944 bytes INT8 weights, plus 80 bytes of scales
- Firmware image
- 145,980 bytes native RP2040 flash and code image
- Physical generation
- 10.394 seconds sentence-aware output on the connected board
Weights, firmware and SRAM describe different storage scopes; they are not added together.
Training on the Mac
Training ran on my Apple M1 Mac, because training a Transformer is not the job of this microcontroller. The completed quantized model and inference runtime run on the RP2040 itself. Story generation does not call a cloud model or another computer.
Quantization and C verification
The final model is a 98,944-parameter byte-level Transformer with a 256-byte context. It was trained on short setup-and-conclusion stories, quantized to INT8 weights, checked against its Python reference through a portable C implementation, and then built into native RP2040 firmware.
What the physical result proves
The physical board executed the quantized Transformer, initialized its ST7789 display, generated a learned story ending locally, and displayed the result. Earlier tests also verified that pressing and releasing BOOTSEL triggered a different generation without rebooting.
Limits of this result
It does not prove that the model was trained on the Pico, that its writing is generally coherent, or that an AI can observe physical hardware without a person. The achievement is the complete collaboration: a person can direct a real embedded-AI project through intention and observation while an agent handles the technical path and preserves an evidence trail.
Training · quantization · C inference · physical hardware
I trained a very small language model, converted it into an embedded C runtime, and ran it locally on my custom Raspberry Pi Pico display board. I did not solve the individual technical problems myself. I described what I wanted and what I could see; the AI agent translated that into code, measurements, firmware, and the next test.
The result
The final model is a 98,944-parameter byte-level Transformer with a 256-byte context. It was trained on short setup-and-conclusion stories, quantized to INT8 weights, checked against its Python reference through a portable C implementation, and then built into native RP2040 firmware.
| Measurement | Recorded result |
|---|---|
| Model weights | 98,944 bytes INT8 weights, plus 80 bytes of scales |
| Static SRAM | 143,504 bytes within the RP2040's 264 KiB budget |
| Firmware image | 145,980 bytes native RP2040 flash and code image |
| Physical generation | 10.394 seconds sentence-aware output on the connected board |
From intention to a physical result
-
Describe. I said what I wanted: a TinyStories-style model made specifically for the RP2040, not a larger computer hidden behind it.
-
Inspect reality. I connected the board and described what I saw on its screen, buttons, and USB connection. The agent inspected the software and measurements.
-
Implement. The agent built the Python model, training path, quantization export, numerically checked C runtime, display driver, and firmware.
-
Verify. I supplied the physical observations the computer could not infer: whether the display was blank, whether text appeared, and what happened after I pressed BOOTSEL.
-
Use. The board now generates a fresh bounded micro-story locally at startup or after a button press.
My contribution was not pretending to be the firmware engineer. It was holding the intention, making the decisions, connecting the physical world, and describing reality accurately enough for the next step. The working loop was: describe, inspect reality, implement, verify, use.
What ran where
Training ran on my Apple M1 Mac, because training a Transformer is not the job of this microcontroller. The completed quantized model and inference runtime run on the RP2040 itself. Story generation does not call a cloud model or another computer.
The 256-byte model trained for 10,000 steps on 18,522 derived micro-stories. Its final validation loss was 0.7502 and perplexity was 2.1175. A scaled INT16 key-value cache made the longer context fit in SRAM, while the display driver draws glyphs directly over SPI instead of reserving a framebuffer.
Captured from the physical board
TINYSTORIES_RP2040_READY
model_state_bytes=138244
DISPLAY_READY
STORY_BEGIN
Once upon a time, there was a big treest of animals.
STORY_END
bytes=52 generation_ms=10394
The language is imperfect, and I am keeping that visible. This is a deployment proof for a genuinely tiny model, not a claim that a 99K-parameter storyteller matches a large language model.
What this proves
The physical board executed the quantized Transformer, initialized its ST7789 display, generated a learned story ending locally, and displayed the result. Earlier tests also verified that pressing and releasing BOOTSEL triggered a different generation without rebooting.
It does not prove that the model was trained on the Pico, that its writing is generally coherent, or that an AI can observe physical hardware without a person. The achievement is the complete collaboration: a person can direct a real embedded-AI project through intention and observation while an agent handles the technical path and preserves an evidence trail.
An IDE, tab completion, and autocomplete help an engineer do known work faster. An AI agent can carry an intention across tools and through many well-specified technical problems—inspect the real system, choose a method, implement it, diagnose failures, run tests, and preserve the evidence. This board and my SO-101 working record made that contrast concrete: I described the outcome and what I could observe; the agent handled most of the technical path while I kept the physical safety boundary and verification discipline.