SO-101 robot simulator

MuJoCo · camera-guided pick-and-place

A simulated SO-101 arm now finds a red cube through its wrist RGB camera, picks it up, searches for a green target, and places the cube safely inside it.

The visible wrist-camera marker makes the perception loop inspectable: search, see, approach, pick, and place.

From motion to perception

The project began with scripted joint motion: useful for proving that the simulated arm and gripper could complete a physical pick-and-place sequence, but dependent on already knowing where everything was.

The next steps moved that knowledge into the wrist camera. First the controller used camera observations to guide the approach. Then it learned to search actively when the cube or target was outside the current view. If an object is not visible, the arm holds safely instead of continuing on a blind assumption.

Task-session progression

  1. 01
    Scripted motion

    Prove the arm, gripper, and pick-and-place sequence in MuJoCo.

  2. 02
    Camera guidance

    Use the wrist RGB view to guide the cube approach and target placement.

  3. 03
    Active search

    Move the wrist view deliberately until the cube or target becomes visible; hold safely while it is unseen.

  4. 04
    Randomized validation

    Run the complete behavior across 18 hidden upright-cube locations in the verified workspace.

Randomized validation

18 / 18 successful hidden-location runs

Each run placed an upright red cube at a hidden location inside the calibrated reachable zones. The controller had to search from the wrist camera view and complete the pick-and-place task. All 18 runs succeeded.

Interactive simulator controls

  • 6
    Randomize Move the cube to another valid hidden location.
  • 7
    Run Start the camera-guided search and pick-and-place sequence.
  • 8
    Reset Return the simulator to its initial state.
  • 9
    Wrist view Toggle the wrist RGB camera view and its visible marker.

Current verified boundary

What works
A calibrated red cube and green target, both inside verified reachable zones, with the cube upright and the scene running in MuJoCo.
What this does not claim
General object recognition, object-orientation estimation, obstacle planning, or control of a real SO-101 arm.

This is a narrow capability made reliable in simulation—not yet a general robot policy. The useful result is the progression itself: scripted action became camera-guided behavior, then active perception, then repeatable validation.