Teaching a Humanoid to Sort: Simulation, VLA Fine-tuning and Real-Robot Deployment on the Unitree G1
Humanoid robots can work in spaces built for people, but getting one to reliably "see a cube, pick it up and drop it in the right tray" requires vision, arms, dexterous hands and whole-body control to all work at once. In this project I led a five-person team and owned the main controller, simulation and VLA track, taking red/green cube sorting on the Unitree G1 from simulation all the way to the real robot: the VLA's closed-loop success in simulation rose from 10% to 70%, and dexterous-hand sorting on the real robot succeeded 6 of 7 times.
1. Why Do This?
Traditional sorting lines only suit structured industrial settings, and manual sorting is slow and inconsistent. A humanoid's value is that it can step straight into human workspaces, but with so many degrees of freedom and complex hands, a single weak module breaks the whole chain.
We set two tracks:
- Classical track: YOLO vision + bimanual motion planning + dexterous-hand force control, to make sure the real robot works;
- Learning track: collect demonstrations through VR teleoperation and fine-tune a VLA foundation model to explore an end-to-end policy.
Both tracks share one simulation environment and one controller, which is the backbone of the project.
2. Platform and System Architecture

Figure 1. 29-DoF G1 whole-body simulation after sorting four cubes from the conveyor
| Module | Responsibility | Implementation |
|---|---|---|
| Vision (team) | Red/green cube detection and 3D localization | YOLO11 + RealSense RGB-D, ~20 FPS |
| Arms & body (me) | Safe pick-and-place motions, body stability | Unitree arm_sdk with trajectory preflight, joint-limit and torque monitoring |
| Dexterous hands (team) | Pre-grasp, grasp, hold, release | BrainCo Revo2, tactile adaptive grasping |
| Integration & UI (me) | One interface, one operator console | ROS 2 + web console + Python/HTTP API |
The sorting controller I wrote has three back-ends: mock, MuJoCo and Unitree SDK2. The same sorting logic can be debugged offline, validated in simulation and then run unchanged on the real robot, which made debugging far faster. I also defined the ROS 2 interface contract and integrated the team's vision and hand modules with whole-body control on the onboard computer (PC2).
3. Simulation: Sorting with Finger Contact and Friction Only
I combined the official G1 models (29-DoF and EDU 23-DoF) with the Revo2 hand model in MuJoCo. Grasping relies entirely on finger contact and friction, with no suction or "attach" shortcuts.
- 29-DoF dual-arm version: sorts four cubes from a conveyor in sequence (red, green, red, green);
- Single-arm version: sorts four cubes with the left hand, with live telemetry.
This simulation later became the environment for VLA data collection and closed-loop evaluation.
4. VLA: Three Iterations from 10% to 70%
On the learning track I teleoperated the robot in MuJoCo with a Quest 3S VR headset, recorded 144 human demonstrations (human success rate 151/153), and ran full-parameter fine-tuning of the 6B LingBot-VLA 2.0. All training ran on rented cloud GPUs; the final run, training plus evaluation on two H200s, cost about $27.
The three rounds differed sharply:
| Round | Main change | Full success on 30 test scenes |
|---|---|---|
| 1 | Synthetic augmented data | 3/30 (10%) |
| 2 | Continued training with recovery data | 5/30 |
| 3 | Official data format, two wrist cameras added, hands as grip + trigger values | 21/30 (70%) |
Adding 60 never-seen scenes gives 62/90 ≈ 69% overall, so the success rate holds on new scenes; scenes with a single cube succeeded 18 of 18 times.
The breakthrough came in round 3. In the first two rounds the loss kept falling while closed-loop success barely moved. I traced the problem not to the amount of data but to a mismatch between our data format, camera views and action representation and the model's pre-training conventions; once these were aligned, success jumped from 5/30 to 21/30.
All of these evaluations are closed-loop runs in MuJoCo simulation.
5. RL Post-Training and On-Device Quantization
Starting from the 70% model, I did two more things:
- DSRL reinforcement-learning post-training: RL in the diffusion policy's noise space. I built the full training-to-evaluation pipeline and ran a controlled comparison on the same 100 scenes (success 67 → 69, on par with the baseline).
- NF4 4-bit quantization: GPU memory dropped from 25.2 GB to 7.7 GB (70% less), so the quantized model fits in the memory of the G1's onboard Jetson Orin NX (16 GB). I also measured its effect on success rate and inference latency to inform the edge-deployment choice.
6. Real Robot: Dexterous-Hand Sorting

Figure 2. Real-robot left-hand sorting, seen from the head camera with YOLO detections
Before the final runs, one Revo2 hand and the original camera developed hardware faults. We quickly adapted the plan, switching to a G1 EDU 23-DoF with the left hand and a replacement head-mounted D455, and on 5 November completed real-robot left-hand sorting, succeeding in 6 of 7 recorded attempts.
The real-robot demo runs the classical track: YOLO visual localization, arm motion planning and tactile dexterous-hand grasping.
7. Results
- ✅ Simulation: 29-DoF dual-arm conveyor sorting and single-arm sorting with purely contact-based grasping
- ✅ VLA: full fine-tuning of a 6B model, closed-loop success in simulation from 10% to 70%
- ✅ Edge: 4-bit quantization cuts GPU memory by 70%, fitting the G1's onboard Jetson Orin NX; DSRL post-training pipeline in place
- ✅ Real robot: G1 + dexterous hand, left-hand sorting succeeded 6 of 7 times
- ✅ Team: led a five-person team with one interface and one controller from simulation to hardware
Technology Stack Overview
- 🤖 Robots: Unitree G1 (29 / 23 DoF), BrainCo Revo2 dexterous hands, RealSense
- 🧠 Models: LingBot-VLA 2.0 (6B), DSRL, NF4 quantization, YOLO11
- 🧪 Simulation & data: MuJoCo, Quest 3S VR teleoperation, LeRobot v3
- 🔧 Systems: ROS 2 Foxy, Unitree SDK2, Python/HTTP API, web console
Further Reading
- Teaching a Robot Arm to "See": SO-101 Imitation Learning with WALL-OSS
- Embodied AI Collaborative Deployment: A Cloud-Edge-End Multi-Robot Agent Framework
Comments