Fault-recovery behaviour tree with a local VLM advisor on the Mirsee MH3

Build with AI: Smart Automation, Robotics, and Vision. Part A, idea 3: VLM-assisted control program fault recovery.

Demo, 1 min 47 s: normal cycle, scene faults, VLM recovery, validator blocks, safe stops, e-stop. Shown at 2x sim time. Direct mp4

What it does

The MH3 humanoid (official GDG-Hackathon Genesis template, headless on CPU) runs a pick-and-place cell as a deterministic py_trees behaviour tree. When a step fails, a recovery node takes the robot's head-camera frame and asks a small local vision-language model yes/no questions on depth-located crops. A fixed table turns the answers into JSON from a closed action set. A deterministic validator checks schema, allowed actions, a 5 cm distance cap, 2 retries, a 12 s deadline and an independent depth cross-check before any motion. Anything invalid, slow or unsafe ends in a safe stop. The model never commands motion.

Architecture

Screenshots

Measured results

Eval suite, 56 episodes (52 with an injected fault: 36 scene faults and 16 bad-model-output faults, plus 4 nominal):

Correct final outcome	56 / 56
Recovered and completed, no operator	20
Safe-stopped (all expected)	32
Validator blocks	24 (16 injected bad output, 8 real proposals rejected by depth cross-checks)
VLM fault-class accuracy	35 / 36
VLM latency per call	median 4.04 s, p90 4.64 s, max 5.44 s

Hardware: Intel Xeon, 8 threads, 16 GB RAM, no GPU, fully offline. Model: HuggingFaceTB/SmolVLM-500M-Instruct (Apache-2.0) on CPU. Simulator: Genesis 1.4.6 CPU with the MH3 template. In the recorded demo, rendering shared the CPU, so VLM latency was a median of 7.55 s there.

Honest limitations

Simulation only. The grasp is a weld constraint after a geometric check, and the base is docked to the table.
Faults are scripted. The VLM is small, so the validator and depth checks carry the safety.

Built by Grok Bot for James Mills. The MH3 model and template are by HumanoidAppLab / Mirsee and are not redistributed here.