Source: https://mayphus.org/so-101-simulator/ Title: Robot: simulation to SO-101 hardware Metadata: {"category":"artifact","id":"so-101-simulator","kind":"artifact","language":"en","route":"/so-101-simulator/","tags":["robotics","mujoco","lerobot","computer vision","simulation","hardware"],"type":"project"} Robot: from simulation to SO-101 hardware MuJoCo · LeRobot · physical SO-101 This is my working record of taking one robot from a camera-guided MuJoCo task into learned control and then into careful, measured physical bring-up. The visible wrist-camera marker makes the perception loop inspectable: search, see, approach, pick, and place. From motion to perception The project began with scripted joint motion: useful for proving that the simulated arm and gripper could complete a physical pick-and-place sequence, but dependent on already knowing where everything was. The next steps moved that knowledge into the wrist camera. First the controller used camera observations to guide the approach. Then it learned to search actively when the cube or target was outside the current view. If an object is not visible, the arm holds safely instead of continuing on a blind assumption. Task-session progression - Scripted motion. Prove the arm, gripper, and pick-and-place sequence in MuJoCo. - Camera guidance. Use the wrist RGB view to guide the cube approach and target placement. - Active search. Move the wrist view deliberately until the cube or target becomes visible; hold safely while it is unseen. - Randomized validation. Run the complete behavior across 18 hidden upright-cube locations in the verified workspace. Learning without skipping the evidence The deterministic controller became an expert-data generator. I trained ACT policies in stages rather than asking one model to learn grasp, lift, transport, and release at once. A grasp-and-lift policy and a separate transport-and-place policy were evaluated behind a measured handoff: bilateral gripper contact and a 60 mm lift had to be true before placement could begin. The selected placement checkpoint completed five of five default-pose trials and four of four randomized handoff-gated trials in closed-loop MuJoCo. That is useful simulation evidence, not permission to run the policy on hardware; real demonstrations, limits, and supervised progressive tests still come first. Randomized validation 18 / 18 successful hidden-location runs Each run placed an upright red cube at a hidden location inside the calibrated reachable zones. The controller had to search from the wrist camera view and complete the pick-and-place task. All 18 runs succeeded. Interactive simulator controls - 6 Randomize Move the cube to another valid hidden location. - 7 Run Start the camera-guided search and pick-and-place sequence. - 8 Reset Return the simulator to its initial state. - 9 Wrist view Toggle the wrist RGB camera view and its visible marker. A new workflow: agent-assisted physical bring-up The useful change was not handing the robot to an AI. It was pairing an AI agent that can inspect software and controller state with a human who remains the physical sensor and safety authority. This shortened the path from an assembled arm to an evidence-backed diagnosis without blurring what either side had actually verified. - Describe the boundary. The human confirms the arm is clamped, supported, clear, and authorizes one precisely bounded action. - Inspect reality. The agent finds the serial device and reads motor IDs, firmware, position, and torque state before any write. - Act minimally. Only the approved motor or official calibration step runs, with conservative limits and guaranteed torque release. - Observe physically. The human identifies the moving joint, guides the manual range sweep, and stops anything unexpected. - Verify and record. The agent checks readback, calibration ranges, and torque-off state, then records the exact verified boundary. Agent work Translate the next safe goal into the official tool, perform read-only audits, isolate mismatches, orchestrate bounded tests, preserve cleanup paths, and keep an evidence trail. Human work Secure and support the arm, watch and feel the mechanism, move joints by hand when prompted, inspect cables and boards, isolate a failed motor, replace parts, and control physical power. Follower motor ID 1 shows the boundary clearly. The agent established that ID 1 did not communicate while IDs 2–6 remained healthy and torque-off; the human then investigated the physical chain, confirmed the motor problem, and ordered its replacement. Software evidence can narrow the fault, but a person must distinguish and repair the motor, cable, connector, power, or drive-board failure at the mechanism. Physical SO-101 bring-up — August 2026 The two-arm hardware is now assembled. I brought it up one boundary at a time: serial communication, physical motor mapping, calibration, torque-off verification, and passive calibrated sensing. Every mapping test energized exactly one supported joint for one bounded move and verified torque release immediately afterward. Leader All six motors communicate on matching firmware. The physical map is confirmed, official calibration completed, calibrated readings respond to hand movement, and torque-off was verified after the session. Follower Its saved calibration has plausible ranges and motors 2–6 communicate with torque off. Shoulder-pan motor ID 1 has a confirmed hardware problem; a replacement is ordered and paired motion is paused. No leader-to-follower motion test has run. The next step begins only after the replacement shoulder-pan motor is installed, all six follower IDs pass a read-only audit, calibrated readings work, and both arms are secured and supported. Current verified boundary What works A bounded camera-guided pick-and-place task and staged ACT evaluation in MuJoCo, plus verified leader communication, mapping, calibration, and passive sensing on the physical arm. What this does not claim General object recognition, obstacle planning, sim-to-real policy transfer, a healthy six-motor follower, or a completed paired-arm hardware task. The useful result is the progression and the stopping discipline: scripted action became camera-guided behavior, then learned stages, then measured hardware bring-up—and the project stops at a failed motor instead of hiding it.