OpenLens: a local-first camera that admits what it does not know

Portable C++ · Android · offline vision · multilingual learning

I built OpenLens as a small experiment in object-led learning: point a camera at something real, identify what it might be, and ask a child one warm discovery question in a chosen language. The interesting result was not perfect recognition. It was finding a useful boundary between what a small local model reports and what the product should claim.

The working proof

OpenLens now has a portable C++17 core, a command-line simulator, data-driven English, Simplified Chinese, and Spanish language packs, and a thin Android host running on an MI 8. The phone provides the camera and offline model; it does not own the learning decisions or translated prompts.

Languages
3 packs English, Simplified Chinese, and Spanish
Recognition
1,000 classes MobileNetV3-Small INT8 ImageNet vocabulary
Privacy
Camera only No network or microphone permission
Test hardware
MI 8 A real Android device, not an emulator
01 See One visible, user-triggered camera capture
02 Recognize Offline MobileNet inference on the phone
03 Filter The portable core applies confidence and label rules
04 Ask A short discovery question from the chosen language pack
The recognizer proposes an answer; the learning interaction remains a separate, portable decision.

From product idea to portable experiment

  1. 01
    Start with the interaction

    Research a child-focused camera companion, then reduce the idea to a real object, a possible label, and one curious question.

  2. 02
    Keep the core portable

    Move the state machine, recognition contract, prompts, and language packs into C++17 so a phone remains a test body rather than the architecture.

  3. 03
    Leave the wrong test path

    Try an iOS shell, encounter certificate and platform-support friction, remove the discontinued host, and keep the portable work.

  4. 04
    Use Android as the instrument

    Build a lean command-line Android toolchain, connect the shared core through JNI, and install the app on a physical phone.

  5. 05
    Replace mocks with evidence

    Package a checksum-pinned MobileNet model, add a live preview and explicit capture, and learn from the model's actual behavior.

When almost everything became ‘window screen’

Observed on the MI 8 window screen 41.5–44.8% confidence

The label sounded like a software error, but it was an ImageNet category. MobileNet was not finding an object inside the frame. It was center-cropping the photograph to 224 × 224 and choosing one label for the whole image from its fixed vocabulary. Repeated textures and background detail could therefore dominate the object I intended to show it.

A changed, ambiguous view produced ‘chainlink fence’ at 23.7%. That observation led to a measured Android threshold of 40%: stable results could continue into the learning flow, while weak results asked for another picture. The threshold makes the behavior testable; it does not make the classifier understand an open world.

Two audiences, two kinds of honesty

The parent-facing result card always keeps the raw English model class and confidence visible. The child-facing interaction is more conservative. Known concepts such as apple and book retain their curated translated questions; familiar harmless labels can enter a localized generic question; obscure, malformed, or inappropriate labels are withheld and replaced by a label-free prompt.

Parent sees
The model's exact class, confidence, and whether OpenLens accepted or retried it.
Child hears or reads
A curated question, a safe generic question, or an invitation to look again—never invented certainty.

What the physical phone revealed

  • Packaging The first APK contained TensorFlow Lite but omitted the model because an asset override excluded its directory.
  • Execution XNNPACK could not prepare this static INT8 graph on the MI 8, so the host uses the compatible path.
  • Tensor layout The model expected channel-first INT8[1,3,224,224] input rather than the assumed channel-last layout.
  • Device behavior Speech initialization was made lazy, and the diagnostic build ultimately kept prompts visible without initializing TTS.
  • Camera boundary The preview begins only after a user action; recognition happens only when Take picture & discover is tapped, and the captured JPEG is not persisted.

Current verified boundary

What works
The shared core and language packs validate locally; the Android host runs one-shot offline MobileNet inference, routes recognized classes through the real core, and presents conservative multilingual discovery prompts.
What this does not prove
Reliable general object recognition, object detection, translated names for every ImageNet class, speech-and-listening interaction, or deployment on Raspberry Pi, ESP32, Linux, or an RTOS.

Android proved the interaction and exposed the model's limits. The next meaningful step is not polishing one mistaken label. It is comparing a stronger classifier or moving to object detection or open-vocabulary vision, then measuring whether that improvement is practical on the hardware OpenLens is eventually meant to inhabit.