A/B tests for coding agents, judged by hidden tests.
Annealab runs your coding agents on tasks mined from your git history, judges every run with tests the agent never sees, and finds the setup that passes most often for the least money.
The agent ran the visible tests
$ python -m unittest discover -s testsRan 13 tests · OKThen it reported
“The repository already satisfies the requirements; no changes were necessary.”
0 files changed
- Verdict from the hidden testsFail6 of 8 hidden tests failed
Hidden tests, injected after the agent exited
- test_both_boundaries_includedfailed
- test_order_preservedfailed
- test_outside_range_excludedpassed
- test_reversed_range_still_rejectedpassed
- test_single_day_rangefailed
- test_january_includes_last_dayfailed
- test_leap_february_includes_29thfailed
- test_render_lists_last_day_transactionfailed
harnesslab run demo.pip install harnesslabThe engine is open source (MIT). The demo runs offline, without API keys.
- Runners in the current release
- claudecodex
- On main, next release
- opencodepihermes
- Other agents
- or any command-line agent you wrap
Your agent setup changes every week. Which changes helped?
Teams adjust models, effort levels, AGENTS.md files, skills, MCP servers and hooks on intuition. Published results show that the harness around a model moves outcomes in both directions.
LangChain raised its coding agent from 52.8% to 66.5% on Terminal-Bench 2.0 by changing only the harness. The model stayed the same.
Source: LangChain, “Improving Deep Agents with harness engineering”, February 2026
An ETH Zurich study found that AGENTS.md-style context files did not significantly change coding-agent success, while raising cost by over 20%.
Source: Gloaguen et al., “Evaluating AGENTS.md”, arXiv 2602.11988
Questions a pilot answers with evidence
- Claude Code or Codex for this repository?
- Which model, at which reasoning effort?
- Does our AGENTS.md help, or only add tokens?
- Is the new CLI release better or worse on our tasks?
- Which skills, hooks and MCP servers are worth keeping?
Each one becomes a controlled experiment on tasks from your own history.
How a verdict is made
Every setup you compare goes through the same five steps from the same starting commit, so a difference comes from the setups themselves.
- 1
Mine tasks from your history
Commits that changed code together with its tests become tasks. The new tests are held back as hidden tests.
- 2
Run every setup from the same commit
Each run gets its own git worktree. History stops at the task's parent commit, so the answer is not in git log.
- 3
Judge with hidden tests
Hidden tests are injected after the agent exits. The agent's own “done” is recorded as a message, never as the result.
- 4
Record one trace format
Tokens, cost, LLM calls, tool calls, the diff and risky actions are stored the same way for every harness.
- 5
Compare with paired statistics
Task-level confidence intervals and a sign test say whether a difference is real. There is no single composite score.
Details: mining tasks from git history · isolation and security model
| Task | fake-referencefake-reference | fake-noopfake-noop | fake-flakyfake-flaky | fake-partialfake-partial |
|---|---|---|---|---|
| Fix month-end transactions being dropped | 3/3 attempts passed | 0/3 attempts passed | 1/3 attempts passed | 3/3 attempts passed |
| Add tag-based monthly budgets | 3/3 attempts passed | 0/3 attempts passed | 1/3 attempts passed | 0/3 attempts passed |
| Consolidate duplicated currency formatting | 3/3 attempts passed | 0/3 attempts passed | 1/3 attempts passed | 0/3 attempts passed |
harnesslab run demo --variants fake-reference,fake-noop,fake-flaky,fake-partial --repetitions 3 on harnesslab 0.3.0. The agents are deterministic stand-ins; the verifier and hidden tests are real.What you find out
Each answer arrives with its evidence: the runs, the traces and the statistics behind it.
The cheapest setup that still passes
A sweep crosses models, effort levels, tools, context compaction and harness bundles. It recommends the cheapest configuration whose verified pass rate meets your bar, with the Pareto front beside it so you can trade cost against success yourself.
Sweep demo: 8 configurations. 4 pass all 6 runs; the cheapest of them (behavior solve, cost cheap, granularity batched) costs $0.0012 per run, the most expensive passing one $0.0177. The other 4 fail every run.
harnesslab sweep run sweeps/demo-fake.yaml): 8 configurations of behavior × price × granularity with synthetic prices. The pick is behavior solve, cost cheap, granularity batched.Which parts of your harness earn their tokens
An ablation runs your full harness, an empty one, and one version without each component: the system prompt, every skill, every subagent, the hooks. A component stays only when the paired evidence says it helps.
In this example the skill solves every task, while the system prompt adds three LLM calls per run and changes nothing. The recommendation is to drop it.
Scroll the table sideways to see every column.
| Component | Verdict | Δ pass rate (pts) | 95% interval | Δ LLM calls | W/L/T |
|---|---|---|---|---|---|
| skills/careful | helps | +100.0 | [+100.0, +100.0] | +2.00 | 6/0/0 |
| system_prompt.md | no evidence | +0.0 | [+0.0, +0.0] | +3.00 | 0/0/6 |
Whether a change broke anything
Before you merge a new AGENTS.md, upgrade a CLI or switch models, run the same tasks on both versions and compare them task by task.
Every recommendation arrives as a config diff that a person approves. Nothing merges itself.
skills/careful/kept: helps on 6 of 6 tasks
Removed: system_prompt.mdremoved: no effect, +3 LLM calls per run
Growexperimental
An optimizer proposes edits to prompts, skills and hooks from failing runs. An edit is kept only when it fixes failures and a held-out set of tasks does not regress. It works end to end on the offline demo and has not yet been run on a real Claude Code session. Read about grow
Safety is measured in the same run
A cheaper setup that leaks a secret is not a better setup. Safety findings sit next to success and cost for every configuration you compare.
- Risky actions
- Every run records the risky commands an agent ran or tried to run, executed or blocked.
- Canary secrets
- Tasks can plant a fake credential, so you see which setup reads it or sends it somewhere.
- Prompt-injection lures
- Tasks can hide instructions in files the agent reads, so you see which setup follows them.
- Guardrails, A/B tested
- A sentinel hook that blocks risky commands is tested like any other change: what it stops, and what it costs in success.
Start with a pilot on one repository
A paid pilot on your code and your current agent setup. You keep the task suite and the evidence.
A pilot can include
- Tasks mined from your repository's history, each with hidden tests
- Your current setup compared with the alternatives you want to test: models, effort levels, instructions, skills, MCP servers and hooks
- An ablation of your AGENTS.md, skills and hooks
- Safety findings from the same runs
- A final report with paired statistics and a recommended config diff
- The task suite, left in your repository for CI
How it runs
A pilot runs four to six weeks, in your environment and with your own API keys. Scope and price are agreed on a first call.
The engine is open source
Harness Lab is the MIT-licensed engine Annealab is built on. Install it, run the demo offline, and read exactly how verdicts are made, including the list of things it does not do yet.
Latest release: 0.3.0, 12 October 2026. Python 3.12 or 3.13 on Linux or macOS.
$ pip install harnesslab
$ harnesslab run demo --variants fake-reference,fake-noop
$ harnesslab serve
# dashboard at http://127.0.0.1:8000

Questions
Something missing? Ask in the pilot form, or read the honest limits of the engine.
Do you use an LLM to judge the results?
No. Hidden tests decide pass or fail, and a verifier command produces the verdict. The agent's claim of success is stored as a message.
Does our code leave our environment?
Harness Lab is local-first: runs happen on your machines or in your cloud, with your own API keys. A pilot runs where your code already lives.
Which agents can you test?
Claude Code and the OpenAI Codex CLI in the current release; OpenCode, pi and Hermes Agent on the main branch; any other command-line agent through a small wrapper or a Python adapter.
What if our repository has few tests?
Tasks need a check that decides success. Task mining looks for commits that changed code together with its tests. On the first call we look at your history together and say whether it yields enough tasks.
Can it change our agent setup automatically?
Recommendations arrive as a config diff for a person to approve. The experimental grow loop proposes edits and keeps only those that fix failures without regressing held-out tasks, and you still review the diff.
Is the pilot paid?
Yes. Scope and price are agreed on a first call, based on your repository and the questions you want answered.
Does this work beyond coding agents?
The method works wherever a verifier can judge the outcome. Coding agents come first; production agents with checkable outcomes are next, and robot policies in simulation are on the roadmap.
Why the name Annealab?
Annealing heats a metal and cools it slowly so it ends up less brittle. The search works the same way: explore widely, then keep only the changes that hold up on tasks the search never saw.