A TinyStories model on my RP2040 board

Training · quantization · C inference · physical hardware

I trained a very small language model, converted it into an embedded C runtime, and ran it locally on my custom Raspberry Pi Pico display board. I did not solve the individual technical problems myself. I described what I wanted and what I could see; the AI agent translated that into code, measurements, firmware, and the next test.

The result

The final model is a 98,944-parameter byte-level Transformer with a 256-byte context. It was trained on short setup-and-conclusion stories, quantized to INT8 weights, checked against its Python reference through a portable C implementation, and then built into native RP2040 firmware.

Model weights
98,944 bytes INT8 weights, plus 80 bytes of scales
Static SRAM
143,504 bytes within the RP2040's 264 KiB budget
Firmware image
145,980 bytes native RP2040 flash and code image
Physical generation
10.394 seconds sentence-aware output on the connected board

From intention to a physical result

  1. 01
    Describe

    I said what I wanted: a TinyStories-style model made specifically for the RP2040, not a larger computer hidden behind it.

  2. 02
    Inspect reality

    I connected the board and described what I saw on its screen, buttons, and USB connection. The agent inspected the software and measurements.

  3. 03
    Implement

    The agent built the Python model, training path, quantization export, numerically checked C runtime, display driver, and firmware.

  4. 04
    Verify

    I supplied the physical observations the computer could not infer: whether the display was blank, whether text appeared, and what happened after I pressed BOOTSEL.

  5. 05
    Use

    The board now generates a fresh bounded micro-story locally at startup or after a button press.

My contribution was not pretending to be the firmware engineer. It was holding the intention, making the decisions, connecting the physical world, and describing reality accurately enough for the next step. The working loop was: describe, inspect reality, implement, verify, use.

What ran where

Training ran on my Apple M1 Mac, because training a Transformer is not the job of this microcontroller. The completed quantized model and inference runtime run on the RP2040 itself. Story generation does not call a cloud model or another computer.

The 256-byte model trained for 10,000 steps on 18,522 derived micro-stories. Its final validation loss was 0.7502 and perplexity was 2.1175. A scaled INT16 key-value cache made the longer context fit in SRAM, while the display driver draws glyphs directly over SPI instead of reserving a framebuffer.

Captured from the physical board

TINYSTORIES_RP2040_READY
model_state_bytes=138244
DISPLAY_READY
STORY_BEGIN
Once upon a time, there was a big treest of animals.
STORY_END
bytes=52 generation_ms=10394

The language is imperfect, and I am keeping that visible. This is a deployment proof for a genuinely tiny model, not a claim that a 99K-parameter storyteller matches a large language model.

What this proves

The physical board executed the quantized Transformer, initialized its ST7789 display, generated a learned story ending locally, and displayed the result. Earlier tests also verified that pressing and releasing BOOTSEL triggered a different generation without rebooting.

It does not prove that the model was trained on the Pico, that its writing is generally coherent, or that an AI can observe physical hardware without a person. The achievement is the complete collaboration: a person can direct a real embedded-AI project through intention and observation while an agent handles the technical path and preserves an evidence trail.