When an LLM Directs a VLA: Deploying and Benchmarking Harness VLA on a Real Franka FR3
VLAs are great at "pick this up and put it there", but given an instruction like "put the vegetables on the table into the blue plate", which needs category understanding and task decomposition, they often grab the wrong thing. A popular idea in 2026 is to let an LLM plan and a VLA execute. In this project I deployed Harness VLA on a real Franka FR3 and compared it with the plain VLA under identical conditions: 14/15 real-robot success versus 11/15.
1. Why Do This?
I have been working on LLM-controlled robots since 2023, starting with a GLM-4V-driven robotic arm. Back then the LLM could only plan, and motions came from hand-written primitives. Today VLAs output continuous actions directly, but semantic understanding and long-task decomposition have become their weak spots.
Harness VLA treats a frozen VLA as a tool orchestrated by an LLM agent: the LLM looks at the images, finds the target and decides when to call the VLA. The paper's quantitative results are all on simulation benchmarks. What I wanted to know was: on a real robot with a different VLA, on which tasks does this hierarchy actually help?
2. Real-Robot Platform and System Setup
| Layer | Component | Role |
|---|---|---|
| Planner | DeepSeek LLM | Reads images, identifies targets, decomposes tasks, picks tools, decides when the task is done |
| Tools | RPent framework | State and image readout, pixel back-projection into robot coordinates, small moves and rotations, gripper control, VLA calls |
| Executor | π0.5 VLA (FR3 checkpoint provided by the lab) | Takes a short instruction and performs a local grasp or placement |
| Hardware | Franka FR3 + Robotiq 2F-85 + wrist and side D405 cameras | Real-world execution and perception |
I deployed the whole system on the real robot on my own: connecting the framework to the model service, integrating both cameras and doing hand-eye calibration. The side camera was calibrated from 32 poses over three sessions, with a leave-one-out error of 0.47 mm / 0.06°.

Figure 1. Hand-eye calibration check for the side D405
3. Evaluation Design: Three Tasks, Five Trials Each
Both methods start from the same home pose and use the same VLA checkpoint:
| Task | Instruction | What it tests |
|---|---|---|
| Task 1 | Put the eggplant in the blue plate | Single object |
| Task 2 | Tennis ball to the blue plate, then eggplant to the white plate | Multiple steps |
| Task 3 | Put only the vegetables in the blue plate (a tennis ball and a white ball are also on the table) | Category understanding |
Plain π0.5 receives the full instruction; in Harness VLA the planner decomposes it and calls the VLA step by step.
4. Results: 14/15 vs 11/15
| Task 1 | Task 2 | Task 3 | Total | |
|---|---|---|---|---|
| Plain π0.5 | 5/5 | 3/5 | 3/5 | 11/15 |
| Harness VLA | 5/5 | 4/5 | 5/5 | 14/15 |
Key findings:
- The biggest gap is on the category task: both failures of plain π0.5 put a ball in the plate as a "vegetable". Harness VLA's planner first identifies the eggplant and moves the arm above it before handing over to the VLA, succeeding 5 of 5 times.
- More reliable on the multi-step task: plain π0.5's failures all came on the second step (the eggplant); with the planner splitting the steps, 4 of 5 succeeded.
- Tied on the single-object task: both reach 5/5. The VLA handles this kind of task on its own, and planner localization only adds time (median 189.7 s vs 46.6 s on successful trials).
Conclusion: hierarchical control clearly pays off on semantic and multi-step tasks; simple tasks are faster to hand straight to the VLA.
5. Results
- ✅ Real-robot deployment: independently brought up "LLM planner + π0.5 VLA" hierarchical control on a Franka FR3, including two-camera integration and hand-eye calibration (0.47 mm / 0.06°)
- ✅ Comparative evaluation: designed a same-condition real-robot comparison of 3 tasks × 5 trials; 14/15 success versus 11/15 for the plain VLA, and 3/5 → 5/5 on the category task
- ✅ Engineering judgment: mapped out where hierarchical control is worth using, and saw on the real robot the planner–VLA feedback issue noted in the paper's Limitations
Technology Stack Overview
- 🦾 Hardware: Franka FR3, Robotiq 2F-85, Intel RealSense D405 × 2
- 🧠 Models: π0.5 VLA, DeepSeek LLM planner
- 🔧 Framework: Harness VLA / RPent, hand-eye calibration, pixel back-projection
Further Reading
- Harness VLA paper
- When Robotic Arms Learn to See and Hear: An Intelligent Robotic Arm System Based on GLM-4V and SAM (LLM-controlled robotic arm, 2023)
- Embodied AI Collaborative Deployment: A Cloud-Edge-End Multi-Robot Agent Framework
Comments