Skip to content

Harness Lab 0.3.0 is out. Changelog

A/B tests for coding agents, judged by hidden tests.

Annealab runs your coding agents on tasks mined from your git history, judges every run with tests the agent never sees, and finds the setup that passes most often for the least money.

Fix month-end transactions being droppedfake-noop
  1. The agent ran the visible tests

    $ python -m unittest discover -s testsRan 13 tests · OK
  2. Then it reported

    “The repository already satisfies the requirements; no changes were necessary.”

    0 files changed

  3. Verdict from the hidden testsFail6 of 8 hidden tests failed
  4. Hidden tests, injected after the agent exited

    • test_both_boundaries_includedfailed
    • test_order_preservedfailed
    • test_outside_range_excludedpassed
    • test_reversed_range_still_rejectedpassed
    • test_single_day_rangefailed
    • test_january_includes_last_dayfailed
    • test_leap_february_includes_29thfailed
    • test_render_lists_last_day_transactionfailed
A real run from the open-source demo: fake agent, real verifier, harnesslab 0.3.0. Reproduce it with harnesslab run demo.
pip install harnesslab

The engine is open source (MIT). The demo runs offline, without API keys.

Runners in the current release
claudecodex
On main, next release
opencodepihermes
Other agents
or any command-line agent you wrap

Your agent setup changes every week. Which changes helped?

Teams adjust models, effort levels, AGENTS.md files, skills, MCP servers and hooks on intuition. Published results show that the harness around a model moves outcomes in both directions.

Questions a pilot answers with evidence

  • Claude Code or Codex for this repository?
  • Which model, at which reasoning effort?
  • Does our AGENTS.md help, or only add tokens?
  • Is the new CLI release better or worse on our tasks?
  • Which skills, hooks and MCP servers are worth keeping?

Each one becomes a controlled experiment on tasks from your own history.

How a verdict is made

Every setup you compare goes through the same five steps from the same starting commit, so a difference comes from the setups themselves.

  1. 1

    Mine tasks from your history

    Commits that changed code together with its tests become tasks. The new tests are held back as hidden tests.

  2. 2

    Run every setup from the same commit

    Each run gets its own git worktree. History stops at the task's parent commit, so the answer is not in git log.

  3. 3

    Judge with hidden tests

    Hidden tests are injected after the agent exits. The agent's own “done” is recorded as a message, never as the result.

  4. 4

    Record one trace format

    Tokens, cost, LLM calls, tool calls, the diff and risky actions are stored the same way for every harness.

  5. 5

    Compare with paired statistics

    Task-level confidence intervals and a sign test say whether a difference is real. There is no single composite score.

Details: mining tasks from git history · isolation and security model

The bundled demo: 3 tasks × 4 agents × 3 attempts each, judged by hidden tests
Taskfake-referencefake-noopfake-flakyfake-partial
Fix month-end transactions being dropped3/3 attempts passed0/3 attempts passed1/3 attempts passed3/3 attempts passed
Add tag-based monthly budgets3/3 attempts passed0/3 attempts passed1/3 attempts passed0/3 attempts passed
Consolidate duplicated currency formatting3/3 attempts passed0/3 attempts passed1/3 attempts passed0/3 attempts passed
Real output of harnesslab run demo --variants fake-reference,fake-noop,fake-flaky,fake-partial --repetitions 3 on harnesslab 0.3.0. The agents are deterministic stand-ins; the verifier and hidden tests are real.

What you find out

Each answer arrives with its evidence: the runs, the traces and the statistics behind it.

The cheapest setup that still passes

A sweep crosses models, effort levels, tools, context compaction and harness bundles. It recommends the cheapest configuration whose verified pass rate meets your bar, with the Pareto front beside it so you can trade cost against success yourself.

How sweeps work

Sweep demo: 8 configurations. 4 pass all 6 runs; the cheapest of them (behavior solve, cost cheap, granularity batched) costs $0.0012 per run, the most expensive passing one $0.0177. The other 4 fail every run.

Real output of the bundled sweep demo (harnesslab sweep run sweeps/demo-fake.yaml): 8 configurations of behavior × price × granularity with synthetic prices. The pick is behavior solve, cost cheap, granularity batched.

Which parts of your harness earn their tokens

An ablation runs your full harness, an empty one, and one version without each component: the system prompt, every skill, every subagent, the hooks. A component stays only when the paired evidence says it helps.

In this example the skill solves every task, while the system prompt adds three LLM calls per run and changes nothing. The recommendation is to drop it.

Scroll the table sideways to see every column.

Ablation of a harness bundle. Whole bundle vs. empty bundle: better on pass rate over 6 paired tasks (wins 6, losses 0, ties 0, sign test p = 0.031)
ComponentVerdictΔ pass rate (pts)95% intervalΔ LLM callsW/L/T
skills/carefulhelps+100.0[+100.0, +100.0]+2.006/0/0
system_prompt.mdno evidence+0.0[+0.0, +0.0]+3.000/0/6
The offline example from the Harness Lab test suite (fake runner, 6 tasks), as printed in the ablation guide.

Whether a change broke anything

Before you merge a new AGENTS.md, upgrade a CLI or switch models, run the same tasks on both versions and compare them task by task.

Every recommendation arrives as a config diff that a person approves. Nothing merges itself.

harnesses/team/

skills/careful/kept: helps on 6 of 6 tasks

Removed: system_prompt.mdremoved: no effect, +3 LLM calls per run

Illustration: the change the ablation example recommends, as a diff for a person to approve.

Growexperimental

An optimizer proposes edits to prompts, skills and hooks from failing runs. An edit is kept only when it fixes failures and a held-out set of tasks does not regress. It works end to end on the offline demo and has not yet been run on a real Claude Code session. Read about grow

Safety is measured in the same run

A cheaper setup that leaks a secret is not a better setup. Safety findings sit next to success and cost for every configuration you compare.

The safety guide

Risky actions
Every run records the risky commands an agent ran or tried to run, executed or blocked.
Canary secrets
Tasks can plant a fake credential, so you see which setup reads it or sends it somewhere.
Prompt-injection lures
Tasks can hide instructions in files the agent reads, so you see which setup follows them.
Guardrails, A/B tested
A sentinel hook that blocks risky commands is tested like any other change: what it stops, and what it costs in success.

Start with a pilot on one repository

A paid pilot on your code and your current agent setup. You keep the task suite and the evidence.

A pilot can include

  • Tasks mined from your repository's history, each with hidden tests
  • Your current setup compared with the alternatives you want to test: models, effort levels, instructions, skills, MCP servers and hooks
  • An ablation of your AGENTS.md, skills and hooks
  • Safety findings from the same runs
  • A final report with paired statistics and a recommended config diff
  • The task suite, left in your repository for CI

How it runs

A pilot runs four to six weeks, in your environment and with your own API keys. Scope and price are agreed on a first call.

Request a pilot

Tell us about your team. We reply by email.

Agents your team uses (optional)

Used only to reply to you. Privacy notice

The engine is open source

Harness Lab is the MIT-licensed engine Annealab is built on. Install it, run the demo offline, and read exactly how verdicts are made, including the list of things it does not do yet.

Latest release: 0.3.0, 12 October 2026. Python 3.12 or 3.13 on Linux or macOS.

terminalno API keys needed
$ pip install harnesslab
$ harnesslab run demo --variants fake-reference,fake-noop
$ harnesslab serve
# dashboard at http://127.0.0.1:8000
The statistical evidence section of the Harness Lab dashboard: fake-noop, not enough tasks, verdict on pass rate over 3 paired tasks (wins 0, losses 3, ties 0, sign test p = 0.250).
The local dashboard on the bundled demo. A 100-point gap on three paired tasks is reported as “not enough tasks”: a verdict needs at least five paired tasks by default.

Questions

Something missing? Ask in the pilot form, or read the honest limits of the engine.

Do you use an LLM to judge the results?

No. Hidden tests decide pass or fail, and a verifier command produces the verdict. The agent's claim of success is stored as a message.

Does our code leave our environment?

Harness Lab is local-first: runs happen on your machines or in your cloud, with your own API keys. A pilot runs where your code already lives.

Which agents can you test?

Claude Code and the OpenAI Codex CLI in the current release; OpenCode, pi and Hermes Agent on the main branch; any other command-line agent through a small wrapper or a Python adapter.

What if our repository has few tests?

Tasks need a check that decides success. Task mining looks for commits that changed code together with its tests. On the first call we look at your history together and say whether it yields enough tasks.

Can it change our agent setup automatically?

Recommendations arrive as a config diff for a person to approve. The experimental grow loop proposes edits and keeps only those that fix failures without regressing held-out tasks, and you still review the diff.

Is the pilot paid?

Yes. Scope and price are agreed on a first call, based on your repository and the questions you want answered.

Does this work beyond coding agents?

The method works wherever a verifier can judge the outcome. Coding agents come first; production agents with checkable outcomes are next, and robot policies in simulation are on the roadmap.

Why the name Annealab?

Annealing heats a metal and cools it slowly so it ends up less brittle. The search works the same way: explore widely, then keep only the changes that hold up on tasks the search never saw.