Source: https://mayphus.org/so-101/ Title: SO-101 Metadata: {"category":"artifact","id":"so-101","kind":"artifact","language":"en","route":"/so-101/","tags":["robotics","mujoco","lerobot","computer vision","simulation","hardware"],"type":"project"} SO-101 MuJoCo · LeRobot · physical SO-101 This is my working record of taking one robot from a camera-guided MuJoCo task into learned control and then into careful, measured physical bring-up. The visible wrist-camera marker makes the perception loop inspectable: search, see, approach, pick, and place. ## From motion to perception The project began with scripted joint motion: useful for proving that the simulated arm and gripper could complete a physical pick-and-place sequence, but dependent on already knowing where everything was. The next steps moved that knowledge into the wrist camera. First the controller used camera observations to guide the approach. Then it learned to search actively when the cube or target was outside the current view. If an object is not visible, the arm holds safely instead of continuing on a blind assumption. ## Task-session progression - **Scripted motion.** Prove the arm, gripper, and pick-and-place sequence in MuJoCo. - **Camera guidance.** Use the wrist RGB view to guide the cube approach and target placement. - **Active search.** Move the wrist view deliberately until the cube or target becomes visible; hold safely while it is unseen. - **Randomized validation.** Run the complete behavior across 18 hidden upright-cube locations in the verified workspace. ## Learning without skipping the evidence The deterministic controller became an expert-data generator. I trained ACT policies in stages rather than asking one model to learn grasp, lift, transport, and release at once. A grasp-and-lift policy and a separate transport-and-place policy were evaluated behind a measured handoff: bilateral gripper contact and a 60 mm lift had to be true before placement could begin. The selected placement checkpoint completed five of five default-pose trials and four of four randomized handoff-gated trials in closed-loop MuJoCo. That is useful simulation evidence, not permission to run the policy on hardware; real demonstrations, limits, and supervised progressive tests still come first. ## Randomized validation 18 / 18 successful hidden-location runs Each run placed an upright red cube at a hidden location inside the calibrated reachable zones. The controller had to search from the wrist camera view and complete the pick-and-place task. All 18 runs succeeded. The browser model above is a separate visual joint explorer. It does not run these MuJoCo tasks, evaluate a learned policy or connect to the physical arms. ## A new workflow: agent-assisted physical bring-up The useful change was not handing the robot to an AI. It was pairing an AI agent that can inspect software and controller state with a human who remains the physical sensor and safety authority. This shortened the path from an assembled arm to an evidence-backed diagnosis without blurring what either side had actually verified. - **Describe the boundary.** The human confirms the arm is clamped, supported, clear, and authorizes one precisely bounded action. - **Inspect reality.** The agent finds the serial device and reads motor IDs, firmware, position, and torque state before any write. - **Act minimally.** Only the approved motor or official calibration step runs, with conservative limits and guaranteed torque release. - **Observe physically.** The human identifies the moving joint, guides the manual range sweep, and stops anything unexpected. - **Verify and record.** The agent checks readback, calibration ranges, and torque-off state, then records the exact verified boundary. ## Agent work Translate the next safe goal into the official tool, perform read-only audits, isolate mismatches, orchestrate bounded tests, preserve cleanup paths, and keep an evidence trail. ## Human work Secure and support the arm, watch and feel the mechanism, move joints by hand when prompted, inspect cables and boards, isolate a failed motor, replace parts, and control physical power. Follower motor ID 1 shows the boundary clearly. The agent established that ID 1 did not communicate while IDs 2–6 remained healthy and torque-off; the human then investigated the physical chain, confirmed the motor problem, and subsequently replaced the motor. Software evidence can narrow the fault, but a person must distinguish and repair the motor, cable, connector, power, or drive-board failure at the mechanism. ## Physical SO-101 bring-up — August 2026 The two-arm hardware is now assembled. I brought it up one boundary at a time: serial communication, physical motor mapping, calibration, torque-off verification, and passive calibrated sensing. Every mapping test energized exactly one supported joint for one bounded move and verified torque release immediately afterward. ### Leader All six motors communicate on matching firmware. The physical map is confirmed, official calibration completed, calibrated readings respond to hand movement, and torque-off was verified after the session. ### Follower At that stage, its saved calibration had plausible ranges and motors 2–6 communicated with torque off. Shoulder-pan motor ID 1 had a confirmed hardware problem; a replacement was ordered and paired motion was paused. No leader-to-follower motion test had run at that stage. The planned next step required the replacement shoulder-pan motor to be installed, all six follower IDs to pass a read-only audit, calibrated readings to work, and both arms to be secured and supported. ## Hardware status confirmed on 9 October 2026 The shoulder-pan motor (ID 1) has been replaced. Calibration is complete, and I have confirmed that the arms move normally after replacement. ## Current verified boundary ### What works A bounded camera-guided pick-and-place task and staged ACT evaluation in MuJoCo, plus verified leader communication, mapping, calibration, and passive sensing on the physical arm. The physical arms have since been calibrated and tested for normal movement after the faulty follower motor was replaced. ### What this does not claim General object recognition, obstacle planning, sim-to-real policy transfer, extended hardware reliability, or a completed paired-arm hardware task. The useful result is the progression and the stopping discipline: scripted action became camera-guided behavior, then learned stages, then measured hardware bring-up. The motor replacement, calibration, and normal movement are recorded separately from any claim of successful paired-arm task execution. ## About the browser model The visual explorer uses the official SO-101 follower URDF and visual meshes from [TheRobotStudio/SO-ARM100](https://github.com/TheRobotStudio/SO-ARM100/tree/a758567c3978dfeefe282ede0500085a48fe8f78/Simulation/SO101), pinned to that source revision. The meshes are simplified for display; joint origins, axes and limits come from the URDF. The starting pose uses the source's new-calibration virtual zero. It is not a scan or calibration of my physical arms. Move a joint slider, rotate the view or reset the pose. Angles and the gripper-frame coordinates describe the displayed kinematic model only. There are no dynamics, collision checks, motor commands or hardware connections. In this source the gripper is angular; it is not the LeRobot 0–100 opening mapping. The original simulation and hardware observations above retain their own dates and limits. [Structured model and provenance](https://mayphus.org/so101/model.json) · [Reference pose](https://mayphus.org/so101/reference.png) · [Conversion and attribution](https://mayphus.org/so101/NOTICE.txt) · [Existing model license](https://mayphus.org/so101/LICENSE.txt). The model data also accompanies this record in the existing public document API and MCP fetch response.