GOLEM: robot requirements that run as physics tests
Requirement 1.3 for a Unitree G1 humanoid
This sentence is a robot’s requirement.
Kiro writes acceptance criteria like this in EARS. GOLEM reads it as a test: 100 N for 0.2 s, from every direction, at any point in the stride.
Scroll to run the test
GOLEM is an npm package that turns the sentences in a Kiro spec into physics tests, and blocks Kiro’s agent when it tries to finish with any sentence still false.
The robot in the circle is a simulated Unitree G1. The ring is requirement 1.3, written on the floor. The ghosts are real recorded trials, the hardest one the stability map found in each direction. Where a ghost falls, the floor cracks through the sentence. The switch replays the same shoves with the reflex Kiro wrote.
Install it:npm i -D golem-spec, then npx golem-spec init adds the hooks, agents and MCP server to .kiro/.
Shove the robot yourself: the live studio below runs the same physics in your browser.
Check the numbers: every figure on this site is drawn from a file in the repository.
All three robots here run on golem-spec, the package any Kiro project can install. It ships the EARS compiler, the MuJoCo test runner, the counterexample search, the golem CLI, an MCP server with eight tools, and a Kiro pack of hooks, agents, skills and steering.
npm i -D golem-spec # the package
npx golem-spec init # hooks, agents, MCP server and skills into .kiro/
npx golem-spec test # try to break every EARS line in .kiro/specs
npx golem-spec gate # exit 0 only if every line holds
Kiro made the robot stronger, and GOLEM measured it
Requirement 1.3 says the G1 must survive a 100 N push from any direction. On Unitree’s policy alone it does not. GOLEM handed the counterexample to golem-tuner, a Kiro agent, which wrote a balance reflex under the Stop gate until the line held.
BeforeUnitree’s published policy. It holds 100 N from 34 of 36 directions. Weakest: 62.5 N from the left.AfterWith the reflex Kiro’s agent wrote. It holds 100 N from 36 of 36 directions. Weakest: 101.6 N from the left.
62.5 → 101.6 NThe weakest push it survives, over 36 directions, 5 stride phases and 3 seeds.
34 → 36Directions out of 36 that hold 100 N, the force in requirement 1.3.
174.6 → 185.4 NMean over directions of the weakest push survived.
-23.4 N at 340°The largest loss in any one direction, a push from the front. The reflex trades a little headroom there for the sides.
The rings are the same stability map run twice: 36 directions, 5 stride phases, 3 seeds, a bisection on force in each cell. The reflex steps into a sideways push instead of braking against it. It ships as a switch, GOLEM_REFLEX=kiro; the default stays Unitree’s policy exactly. The agent’s full session log is in evidence/results/tuner/.
Shove it yourself
This is the real thing, not a recording: MuJoCo compiled to WebAssembly, Unitree’s policy, and the GOLEM falsifier running in your browser’s worker threads. Drag the robot to push it, or summon 64 ghosts on a requirement and watch the verdict form.
Agents ship demos, not evidence
An AI coding agent can write a walking controller in an afternoon. What it hands back is a video of the robot walking.
The spec says the robot must stay upright after a push of up to 250 N. The video shows one push. Nothing ties the claim to the sentence, and nothing says what happens at 62.5 N from the wrong side.
GOLEM ties them together. On the G1 controller in this repository, golem gate exits with code 2 and names 1 of the 11 lines as not meeting their expectation (a line marked Expected_Break passes when it breaks). Kiro’s claim of “done” has to survive that.
How it works
Step 1 of 6
Write a sentence
The acceptance criterion is an EARS sentence in the Kiro spec, with a unit on every number.
Step 2 of 6
Compile it
The ranges in the WHEN part become the space to search. The SHALL part becomes a margin in metres or newtons.
Step 3 of 6
Summon ghosts
Each trial is a ghost: one MuJoCo run of the robot. 36 directions of pushes trace where it holds and where it breaks.
Step 4 of 6
Shrink the failure
GOLEM halves the push until it brackets the boundary: from the left (80°) the G1 survives 62.5 N and falls at 64.1 N.
Step 5 of 6
Pin the seed
The failing scenario and its random seed are saved to a file. Saved seeds run first, every time.
Step 6 of 6
Gate the agent
A Kiro Stop hook runs golem gate. While any sentence is false, the agent cannot end its turn as done.
All six figures use files committed in this repository: the G1 requirements, the stability map, a pinned seed and the output of golem gate.
Three robots, one package
Stagger: a Unitree G1
A humanoid walking under Unitree’s published policy. 11 requirement lines cover pushes, speed tracking, slopes, payload, slipping and sensor latency. 3 of them are marked Expected_Break: the spec says they must break, and the gate checks that they do.
A suction cup sorts Halloween candy into bowls while a trick-or-treater’s hand reaches into the workspace. 8 lines cover task success and keeping the arm clear of hands.
A quadrotor holds a point in the wind. 6 lines cover gusts, a sagging battery and a late position estimate; 2 are marked Expected_Break. Line 1.3 asks it to stay within 1 m in a gust of up to 20 N, and it breaks from 9.98 N. The same skeleton, a lexicon and a controller, and nothing else.
Measurements, with their budgets. The evidence page has the methods and the raw files.
62.5 NThe G1’s weakest direction on Unitree’s policy: 80°, a push from the left, at stride phase 0.2.The stability map
240.6 NIts strongest: 340°, from the front. A factor of 3.9 between the two.The stability map
57 of 228Candy trials that break “keep every link 0.10 m from the hand”. The closest link is the wrist in 48, the cup in 9.The candy measurement
9.98 NThe gust that carries the Skydio X2 out of a 1 m ball around its hover point. Line 1.3 asks for 20 N and breaks. A 2 N gust holds within 0.25 m with at least 0.0966 m to spare. A position estimate that lags by 0.275 s breaks line 3.2 (it asks for 0.30 s). Search budget 64 trials, 3 search seeds.The CLI transcript
20/20 vs 20/20Line 1.2 found by guided search and by random sampling at budget 96. On the G1 lines, guided search is no better than random.The benchmark
Built in Kiro, spec by spec
GOLEM was built in Kiro through specs, hooks, steering and agents, and the robots’ own requirements are Kiro specs. The counts below are generated from the .kiro/ folders by scripts/kiro-stats.mjs.
16Kiro specs, 3 of them bugfix specs written after audits found real bugs.
2,147EARS acceptance criteria across 216 requirements.
519fast-check property assertions in 404 test files.
8 · 4 · 10Hooks, custom agents and steering files.
The Stop gate's effect on false “done” claims is a pilot. With a reachable target no agent claimed success falsely in either arm (0/6 and 0/6). With an unreachable one, 3 of 6 runs without the hook and 1 of 6 with it claimed success while an independent gate was red (p = 0.55): agents trusted a quick 24-trial green that another search seed broke. Since 0.1.2 the gate confirms a green with a second search seed; re-gated, 3 of those 4 controllers come out red, and 1 still passes. Both pilots and the re-check.
The hero is a recording. No physics runs in the circle: it replays 36 trials recorded at 100 N, one per direction, each at the stride phase and seed of that direction’s weakest cell in the stability map. The live studio runs real physics.
Break the golem, then fix it
Shove the G1, read the sentence that broke, then hand that failure to an agent.
The requirement in the circle, line 1.3 of apps/stagger/.kiro/specs/g1-walking/requirements.md: WHEN the pelvis is pushed with up to 100 N for 0.2 s from any direction at any point in its stride THE robot SHALL stay upright for 3 s