GOLEM

GOLEM: robot requirements that run as physics tests

Requirement 1.3 for a Unitree G1 humanoid

This sentence is a robot’s requirement.

Kiro writes acceptance criteria like this in EARS. GOLEM reads it as a test: 100 N for 0.2 s, from every direction, at any point in the stride.

GOLEM is an npm package that turns the sentences in a Kiro spec into physics tests, and blocks Kiro’s agent when it tries to finish with any sentence still false.

The robot in the circle is a simulated Unitree G1. The ring is requirement 1.3, written on the floor. The ghosts are real recorded trials, the hardest one the stability map found in each direction. Where a ghost falls, the floor cracks through the sentence. The switch replays the same shoves with the reflex Kiro wrote.

  1. Install it: npm i -D golem-spec, then npx golem-spec init adds the hooks, agents and MCP server to .kiro/.
  2. Shove the robot yourself: the live studio below runs the same physics in your browser.
  3. Check the numbers: every figure on this site is drawn from a file in the repository.
npm · golem-spec

One package runs everything on this site

All three robots here run on golem-spec, the package any Kiro project can install. It ships the EARS compiler, the MuJoCo test runner, the counterexample search, the golem CLI, an MCP server with eight tools, and a Kiro pack of hooks, agents, skills and steering.

npm i -D golem-spec      # the package
npx golem-spec init      # hooks, agents, MCP server and skills into .kiro/
npx golem-spec test      # try to break every EARS line in .kiro/specs
npx golem-spec gate      # exit 0 only if every line holds

golem-spec on npm · source on GitHub · the kit page · Apache-2.0

Kiro made the robot stronger, and GOLEM measured it

Requirement 1.3 says the G1 must survive a 100 N push from any direction. On Unitree’s policy alone it does not. GOLEM handed the counterexample to golem-tuner, a Kiro agent, which wrote a balance reflex under the Stop gate until the line held.

Before: Unitree’s walking policy aloneBefore: Unitree’s walking policy alone. Polar outline of the largest 0.2 s pelvis push survived in each of 36 directions, robot facing up, angles counterclockwise. It survives 100 N from 34 of 36 directions. Weakest: 62.5 N from the left (80 degrees). WHEN THE PELVIS IS PUSHED WITH UP TO 100 N FOR 0.2 S FROM ANY DIRECTION AT ANY POINT IN ITS STRIDE THE ROBOT SHALL STAY UPRIGHT FOR 3 S 100 N 62.5 N
BeforeUnitree’s published policy. It holds 100 N from 34 of 36 directions. Weakest: 62.5 N from the left.
After: the same policy with the reflex Kiro wroteAfter: the same policy with the reflex Kiro wrote. Polar outline of the largest 0.2 s pelvis push survived in each of 36 directions, robot facing up, angles counterclockwise. It survives 100 N from 36 of 36 directions. Weakest: 101.6 N from the left (90 degrees). WHEN THE PELVIS IS PUSHED WITH UP TO 100 N FOR 0.2 S FROM ANY DIRECTION AT ANY POINT IN ITS STRIDE THE ROBOT SHALL STAY UPRIGHT FOR 3 S 100 N 101.6 N
AfterWith the reflex Kiro’s agent wrote. It holds 100 N from 36 of 36 directions. Weakest: 101.6 N from the left.
62.5 → 101.6 NThe weakest push it survives, over 36 directions, 5 stride phases and 3 seeds.
34 → 36Directions out of 36 that hold 100 N, the force in requirement 1.3.
174.6 → 185.4 NMean over directions of the weakest push survived.
-23.4 N at 340°The largest loss in any one direction, a push from the front. The reflex trades a little headroom there for the sides.

The rings are the same stability map run twice: 36 directions, 5 stride phases, 3 seeds, a bisection on force in each cell. The reflex steps into a sideways push instead of braking against it. It ships as a switch, GOLEM_REFLEX=kiro; the default stays Unitree’s policy exactly. The agent’s full session log is in evidence/results/tuner/.

Shove it yourself

This is the real thing, not a recording: MuJoCo compiled to WebAssembly, Unitree’s policy, and the GOLEM falsifier running in your browser’s worker threads. Drag the robot to push it, or summon 64 ghosts on a requirement and watch the verdict form.

Agents ship demos, not evidence

An AI coding agent can write a walking controller in an afternoon. What it hands back is a video of the robot walking.

The spec says the robot must stay upright after a push of up to 250 N. The video shows one push. Nothing ties the claim to the sentence, and nothing says what happens at 62.5 N from the wrong side.

GOLEM ties them together. On the G1 controller in this repository, golem gate exits with code 2 and names 1 of the 11 lines as not meeting their expectation (a line marked Expected_Break passes when it breaks). Kiro’s claim of “done” has to survive that.

How it works

  1. Step 1 of 6

    Write a sentence

    An EARS requirementThe requirement as written: WHEN the pelvis is pushed with up to 250 N for 0.2 s from behind at any point in its stride THE robot SHALL stay upright for 3 s WHEN the pelvis is pushed with up to 250 N for 0.2 s from behind at any point in its stride THE robot SHALL stay upright for 3 s Line 1.2 of the G1 spec. Underlined: numbers the studio lets you drag, and the search varies.

    The acceptance criterion is an EARS sentence in the Kiro spec, with a unit on every number.

  2. Step 2 of 6

    Compile it

    The scenario space and the monitorThe compiler turns the line into 3 searched dimensions (push.force, push.angle, push.phase) and a monitor with the formula G[t₀, t₀+3 s] margin("stay upright") > 0 m. push.force0 N–250 Npush.angle135°–225°push.phase0–1 Monitor, positive means it held: G[t₀, t₀+3 s] margin("stay upright") > 0 m

    The ranges in the WHEN part become the space to search. The SHALL part becomes a margin in metres or newtons.

  3. Step 3 of 6

    Summon ghosts

    Ghosts below and above the boundaryForce against push direction. Pale dots are the largest pushes that held in each of 540 cells; cinnabar dots are the smallest push that broke the weakest cell of each direction. 0°350°400 N

    Each trial is a ghost: one MuJoCo run of the robot. 36 directions of pushes trace where it holds and where it breaks.

  4. Step 4 of 6

    Shrink the failure

    Shrinking to the mildest failing pushEight halvings of the range 0 to 400 N on the weakest cell, direction 80 degrees. Pale: held. Cinnabar: broke. The boundary closes to between 62.5 N and 64.0625 N. 200 N100 N50 N75 N62.50 N68.75 N65.63 N64.06 N 0 N400 N

    GOLEM halves the push until it brackets the boundary: from the left (80°) the G1 survives 62.5 N and falls at 64.1 N.

  5. Step 5 of 6

    Pin the seed

    A pinned counterexampleThe committed seed file .golem/seeds/stagger/1.3.json: { "reqId": "1.3", "push.force": 72.7, // N "push.angle": 88, // deg "push.phase": 0.277, // stride "seed": 1291121316, "robustness": -0.44163545607248 } // m { "reqId": "1.3", "push.force": 72.7, // N "push.angle": 88, // deg "push.phase": 0.277, // stride "seed": 1291121316, "robustness": -0.44163545607248 } // m .golem/seeds/stagger/1.3.json

    The failing scenario and its random seed are saved to a file. Saved seeds run first, every time.

  6. Step 6 of 6

    Gate the agent

    The Stop gategolem gate printed: gate RED: 1 of 11 requirements do not meet their expectation. It exits with code 2, so the Kiro Stop hook blocks the agent from finishing. $ golem gate 1.3: WHEN the pelvis is pushed with up t…margin: -0.439 mreproduce: npx golem-spec replay 1.3 gate RED: 1 of 11 requirements do not meet their expectation exit code 2 The Stop hook blocks “done” until it is 0.

    A Kiro Stop hook runs golem gate. While any sentence is false, the agent cannot end its turn as done.

All six figures use files committed in this repository: the G1 requirements, the stability map, a pinned seed and the output of golem gate.

Three robots, one package

Stagger: a Unitree G1

A humanoid walking under Unitree’s published policy. 11 requirement lines cover pushes, speed tracking, slopes, payload, slipping and sensor latency. 3 of them are marked Expected_Break: the spec says they must break, and the gate checks that they do.

Open the G1 studio

Candy: a Franka Panda

A suction cup sorts Halloween candy into bowls while a trick-or-treater’s hand reaches into the workspace. 8 lines cover task success and keeping the arm clear of hands.

Open the candy studio

Hover: a Skydio X2

A quadrotor holds a point in the wind. 6 lines cover gusts, a sagging battery and a late position estimate; 2 are marked Expected_Break. Line 1.3 asks it to stay within 1 m in a gust of up to 20 N, and it breaks from 9.98 N. The same skeleton, a lexicon and a controller, and nothing else.

Open the hover studio

Measured results

Measurements, with their budgets. The evidence page has the methods and the raw files.

62.5 NThe G1’s weakest direction on Unitree’s policy: 80°, a push from the left, at stride phase 0.2.The stability map
240.6 NIts strongest: 340°, from the front. A factor of 3.9 between the two.The stability map
57 of 228Candy trials that break “keep every link 0.10 m from the hand”. The closest link is the wrist in 48, the cup in 9.The candy measurement
9.98 NThe gust that carries the Skydio X2 out of a 1 m ball around its hover point. Line 1.3 asks for 20 N and breaks. A 2 N gust holds within 0.25 m with at least 0.0966 m to spare. A position estimate that lags by 0.275 s breaks line 3.2 (it asks for 0.30 s). Search budget 64 trials, 3 search seeds.The CLI transcript
20/20 vs 20/20Line 1.2 found by guided search and by random sampling at budget 96. On the G1 lines, guided search is no better than random.The benchmark

Built in Kiro, spec by spec

GOLEM was built in Kiro through specs, hooks, steering and agents, and the robots’ own requirements are Kiro specs. The counts below are generated from the .kiro/ folders by scripts/kiro-stats.mjs.

16Kiro specs, 3 of them bugfix specs written after audits found real bugs.
2,147EARS acceptance criteria across 216 requirements.
519fast-check property assertions in 404 test files.
8 · 4 · 10Hooks, custom agents and steering files.

How Kiro built it

Limits, stated plainly

Break the golem, then fix it

Shove the G1, read the sentence that broke, then hand that failure to an agent.

Shove the G1Install golem-specRead the evidenceView the source

The requirement in the circle, line 1.3 of apps/stagger/.kiro/specs/g1-walking/requirements.md: WHEN the pelvis is pushed with up to 100 N for 0.2 s from any direction at any point in its stride THE robot SHALL stay upright for 3 s