▸ I Taught Two Robot Arms to Set a Dinner Table. The Hard Part Was Proving Tt

I spent seven days building a robot that does not exist.

Not a metaphor. The AI Infra Summit Hackathon ran online, with no hardware in the room. Two robot arms, one dinner table, one spoken command — all of it inside a physics engine. I had never written a line of robotics code before this.

Here is what I built, what broke, and the bug that taught me the most.

The hackathon and the theme

AI INFRA HACKATHON, online, Intel-themed. The brief: build a bimanual VLA - a Vision Language Action system that drives two arms - and run the inference on Intel AI PC hardware.

The machine was an Intel Core Ultra 7 358H, codenamed Panther Lake, from the Core Ultra Series 3 line. I had remote access to one instance:

• 1 socket, 8-16 cores
• 32-64 GB memory
• 512-1024 GB disk
• Arc B390 integrated GPU and an NPU5010 neural accelerator

The task: make two SO-101 robot arms work together to arrange a dinner table. Software only. The simulation ran in MuJoCo, a physics engine.

The interesting constraint was not the language. It was geometry. Two arms, five usable joints each, reaching into the same small volume above one table.

https://github.com/Sumit-Dwivedi/intel-bimanual-vla/blob/master/docs/images/m02-scene.png — the full scene, both arms and the table. Two SO-101 arms, a plate, a fork, a spoon, a mug and a water bottle. Everything that follows happens on this table.

What the system does

A spoken command travels the whole path:

1. Speechmatics transcribes the audio.
2. A rule-based grammar turns the words into a plan. It has 43 tests.
3. A scripted controller executes the plan in MuJoCo.
4. OpenVINO runs the perception model on Intel silicon.

Four skills work from end to end:

• pick(fork)
• place(fork)
• pick(water_bottle)
• handoff(arm A to arm B, fork)

One command runs all four in about 25 seconds.

https://github.com/Sumit-Dwivedi/intel-bimanual-vla/blob/master/docs/images/m06-handoff-candidate-1.png The handoff. Arm B holds the fork, arm A has let go and moved clear. Both facts are checked in code, not judged by eye.

The voice add-on

Voice was not in the original plan. It was a bonus track I picked up once the four skills worked.

The whole thing talks to the Speechmatics batch API using Python's standard library alone. No SDK, no extra package, no second environment. It slots in as a second implementation of a CommandSource interface I had already built for typed text, so nothing downstream of that boundary changed - the grammar, the skills and the demo runner never learn that the words arrived as audio.

I had no microphone, so I synthesized the test clip with the speech engine built into Windows. The phrase was "Give the fork to arm B".

python run_demo.py --voice data/voice/give_fork_to_arm_b.wav

Speechmatics returned that transcript at 0.983 confidence. The grammar turned it into handoff(arm=B, target=fork, from_arm=A). The controller ran it and passed, in about 20.6 seconds.

One detail cost me a while. The standard recognition model kept mishearing the arm name - a single letter at the end of a sentence is a hard target. Moving to their enhanced model fixed it, and that setting is now pinned in the code with a note explaining why.

Here is the part I liked.The voice-driven handoff reported 6610 frames and 0.1946 m of final arm separation. The typed-command run reports the same two numbers, exactly. The audio path is a genuine front end and not a parallel implementation, and those matching numbers are how I know.

Intel silicon: the numbers

I trained a small pose-estimation network, PoseNet, on the Arc B390 iGPU. It reaches 2.6-3.2 mm mean absolute error. Then I exported it to OpenVINO and measured it on every device the chip offers.

At batch 16 the GPU reaches 5800 Hz. Latency grows more slowly than batch size on all three devices, so throughput keeps climbing.

The iGPU beat the NPU here. That surprised me. The model is small, and the NPU pays a fixed cost per call that the GPU does not.

The problems

This is the part worth reading.

The arm has a joint that does nothing

The SO-101 has six actuators. Only five of them move the gripper. The sixth drives the jaw. I spent most of a day solving for six degrees of freedom before I understood that I had five. A five-joint arm cannot satisfy an arbitrary position and orientation, and that single fact shaped every decision afterward.

MuJoCo refused to load on my laptop

Windows Application Control blocks the unsigned MuJoCo library. The error is opaque:

ImportError: DLL load failed while importing _specs:
An Application Control policy has blocked this file.

So every simulation ran on the remote Intel box, over an SSH jump host. Later the block appeared, vanished, and appeared again on the same machine in the same day. Never trust an environment that worked yesterday.

The arms kept colliding

The handoff failed for days. Arm B would reach for the fork and hit arm A. I tried moving the arms apart, staging the approach, changing the geometry. Nothing held.

The fix was not geometric. The idle arm was drifting, because I held it in place by chasing its current position every step. Freezing one control snapshot and rewriting that same snapshot each step cut the drift to 0.000774 radians. The collision disappeared with it.

I measured a path and missed the collision in it

I sampled a motion path at 21 points and declared it clear. It was not. The collision sat at 87% along the path, between two of my samples. A 101-point sweep found it: 2.69 cm of penetration.

A sampling interval is a claim about what cannot happen between samples. Mine was wrong, and nothing in the result told me so.

The bug that taught me the most

My demo passed. Four skills, every check green. Then I recorded a 40-second video and watched it properly.

The mug fell over. The water bottle rolled off the table.

Nothing was broken. My success criteria only ever asked one question: did the fork end up where I wanted? They never asked what happened to everything else.

So I wrote a second measuring tool. It checks five things, not one:

1. The target object reached its destination.
2. No other object moved more than 5 mm.
3. No arm touched any other object.
4. The two arms never touched each other.
5. Peak joint speed stayed below a set limit.

Then I scored my own working demo with it.

It passed 0 of 3.

The place skill shoved the mug 63.2 mm. The handoff spent 3179 of its 6610 steps - 48% of its runtime - with the two arms inside each other, and reported success every time. Peak joint speed hit 6.937 rad/s, which is why things flew.

https://github.com/Sumit-Dwivedi/intel-bimanual-vla/blob/master/docs/videos/full-sequence-demo.mp4 The demo that passes every check. Watch the mug and the bottle.

The rebuild

I replaced the motion stack on a branch. Closed-form top-down inverse kinematics, and smooth cubic-spline joint trajectories instead of direct position commands.

The new stack lowered peak joint velocity by 11.9 times and peak acceleration by 21 times, and reached the same final accuracy. It passes 2 of 3 individual skills on all five criteria. A six-stage bimanual relay passes all five criteria at every stage, and nothing else on the table moves.

https://github.com/Sumit-Dwivedi/intel-bimanual-vla/blob/master/docs/videos/v2-relay-demo.mp4 The rebuilt stack. The fork travels from one arm to the other by way of the table, and everything else stays put.

It also lost things. The rebuilt stack cannot reach the water bottle at all. And it has no working direct hand-to-hand transfer - the relay hands the fork over by putting it down. I diagnosed that failure and wrote down the fix. I did not solve it.

Two shortcuts I turned down

Twice I found a change that turned a failing test into a passing one.

The first widened a grasp tolerance from 5 cm to 12 cm. The second widened it again to 11.5 cm for the handoff, and the handoff then reported success.

I checked what the second one actually did. The receiving gripper never touched the fork at any point in the run. The object attached across a 10.8 cm gap. That is a teleport wearing the costume of a handoff.

Both changes were measured. Both were reverted.

A passing number you cannot explain is worth less than a failure you can.

Numbers, with the qualifier that makes them true

• pick(fork) and place(fork): 20 of 20 seeds.
• pick(water_bottle): 9 of 20, or 45%. An earlier 10-seed run gave 6 of 10. The small sample was optimistic, so I report the larger one.
• handoff: 20 of 20, and that figure is degenerate. Its test envelope is a single point, so every seed runs the identical scenario. The number measures determinism, not robustness.

That last one matters. I could have printed "handoff: 20/20" and stopped. It would have been true and useless.

What did not work

Four skill and object combinations fail, each with a measured reason and a numeric residual. pour never got built, so the brief's own example command does not run from end to end. The test suite reports 66 passed and 4 failed.

All of it is written down as instructed to claude. 59 architecture decision records cover every choice and every reversal, including the two rebuilds I abandoned.

Code and links

Everything is public - the code, the videos, the benchmarks, the decision records, and the failures.

GitHub: https://github.com/Sumit-Dwivedi/intel-bimanual-vla

Clone it, install the pinned requirements, and run one command. The repository holds both demo videos, the per-device OpenVINO benchmarks, the 20-seed evaluation, and eight learning notes I wrote for a reader with no robotics background - which is what I was when I started.

I did not build a robot that sets a dinner table. I built one that sets most of a dinner table, and I can tell you exactly which parts fail and why.

That turned out to be the more useful thing.