AgentHippo
An agentic system’s behavior is set by a handful of discrete choices: the model each agent runs on, the prompts and skills that instruct it, the harness that executes it (Claude Code, Codex or a custom one), how many agents there are and how they hand work to each other, and what each one remembers. We call these choices the system’s parameters, θ. Tuning θ resembles training a model, with three differences: there is no gradient, every evaluation costs real money and hours, and the search space is a set of edits rather than a vector of numbers. We do not fine-tune any model weights in this work.
We study this problem on accounts-payable (AP) recovery. Given a folder of a company’s supplier contracts, invoices and ERP exports, the system must report every amount the company is owed back: a price above the contracted rate, an invoice paid twice, a credit never applied, a rebate never claimed. Each finding must give its amount to the cent and the document line that proves it, and the system must report nothing false. Our starting point was a multi-agent workflow on gpt-6.1-sol, with an onboarding agent, a reader, a second reader and a refuter.
We describe a tuning loop in which a coding agent (Claude Code or Codex) proposes changes to θ, deploys them on budget-capped VMs, scores them against synthetic businesses with exact answer keys, and analyzes every error. Researchers choose the directions and decide which changes to keep. Run on AP, the loop found that:
- A single agent that keeps a casebook of a reviewer’s verdicts matches the multi-agent workflow’s accuracy at the same cost on AP and at a quarter to a third of the cost on other industries.
- Haiku 5.5 with that memory beats Opus 5.5 without it: 0–3 false findings per drop against 6–46, at about a seventh of the cost.
- What the agent learns from one review carries over to later drops in 17 of 18 repeats.
Our stack
The loop has two layers, and this is the stack we used:
- Claude Code is the research assistant agent. It runs the loop: it edits packs and workflow files, deploys them, runs the benchmarks, scores the results, reads the logs and proposes the next change. You could use Codex or another coding agent in this role instead.
- AgentHippo is the runtime underneath. It provides the signed, versioned agent packs,
agenthippo serveto deploy them on any VM, the control plane that holds every deployment’s budget cap and stop switch, and the gateway and Spotlight that log every model call. It runs the agents being tuned, whichever harness and model they use.
Claude Code and Codex also appear a second time, as the two agent harnesses we evaluate. Inside a pack, the harness is a setting (engine: claude or engine: codex; AgentHippo also loads custom engines). So the harness is one of the parameters in θ, and we compare the two on the same worlds. To keep the roles apart, “the research assistant agent” below always means the Claude Code session running the loop, and “engine” means the harness inside a pack under test.
Setup
Data. Each dataset is a synthetic business (“world”) written by a seeded generator: a folder of several hundred documents and a hidden answer key that is exact to the cent, plus look-alikes that must not fire. Four worlds are AP businesses: a lighting-fixture assembler (A), a millwork manufacturer (B), an auto dealership (C) and a property manager (D), each with 28–31 findings. Two worlds from other industries check that we are not tuning to AP: a family-law firm (T2) and a Vietnamese loan-underwriting book (G). Documents arrive in drops. Drop 0 is the first folder, and later drops are new months of documents for the same business. Development seeds serve as the training set. Sealed seeds, generated after a candidate is frozen, are the evaluation set. A leak gate fails the build if anything an agent can see is copied from a world or its key.
Metric. Plain-code scorers that never change report findings as found / false / missed, the share of amounts exact to the cent, and the cost of every run.
Parameters. For AP, θ covers:
- the model: gpt-6.1-sol, or Haiku, Sonnet or Opus 5.5;
- the harness: Codex or Claude Code;
- each agent’s prompt and skills;
- the topology: multi-agent workflows or a single agent;
- the memory: none, a casebook of the reviewer’s verdicts, or the AP standard (the conventions an AP auditor already knows).
The tuning loop
┌─────────────────────┐ ┌─────────────────────┐
│ 1 PROPOSE │ │ 2 RUN │
│ a change to θ: ├─────────►│ sign packs, deploy │
│ prompt, model, │ │ on VMs, run the │
│ agents, workflow │ │ workflow on the │
│ (Claude Code) │ │ training worlds │
└──────────▲──────────┘ └──────────┬──────────┘
│ │
│ tuning θ, no weights │
│ ▼
┌──────────┴──────────┐ ┌─────────────────────┐
│ 4 DECIDE │ │ 3 EVALUATE │
│ keep or drop the │◄─────────┤ score against the │
│ change, choose the │ │ answer keys, read │
│ next direction │ │ the logs of every │
│ (researchers) │ │ miss (Claude Code) │
└─────────────────────┘ └─────────────────────┘Here is one turn of the loop on AP. In step 3, Claude Code read the spend logs and found that the refuter accounted for about 80% of an AP night’s cost, while the rules it was checking were already vetted. The researchers chose a direction in step 4: check the agent with code instead of with another agent. In the next step 1, Claude Code proposed a single agent that writes its own parser and rules, keeps the reviewer’s verdicts as test cases, and must reproduce every settled case before it reports. Step 2 deployed that agent on the AP worlds and step 3 scored it. The results below came out of turns like this one.
Before each run, Claude Code writes down the change, the expected result and a stop rule. A change has to help across worlds; one that helps a single world counts as overfitting and is not kept.
What the loop needs
- Versioned, reproducible packs. Each agent’s prompt, skills, tools, engine and model live in one signed pack. refuter 0.5.20 or onboarder 0.9.33 refers to exactly one configuration, so a result can be rerun a month later. Moving the AP reader from Codex to Claude Code, or from gpt-6.1-sol to Haiku, is a setting on the pack.
- Every input and output logged. All calls go through the gateway into Spotlight. When an invoice is missed, Claude Code reads what the reader actually saw and wrote.
- A budget cap on every deployment, set outside the pack. The agent being tuned cannot raise its own limit, so the loop can run overnight without a surprise bill.
- Workflows as files, not a workflow platform. Airflow and similar tools are built for workflows you have already settled on. While you are still searching, Claude Code writes each workflow as a small declarative file (code checks, agent steps, a person’s decision, and where each failure goes back to), and a separate pack draws every run. Changing the AP workflow from four agents to one is a diff.
Many agents, many machines, one control plane
The agents under test do not run on one machine. Our campaigns ran each pack as its own deployment, spread over several VMs and a laptop, all started with agenthippo serve.
┌──────────────── control plane ────────────────┐
│ a lease per deployment: budget cap, │
│ version pin, stop, rollback (changed │
│ only by a signed-in person) │
└──────▲────────────────▲────────────────▲──────┘
│ poll lease │ │
┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
│ VM A │ │ VM B │ │ laptop │
│ reader pack │ │ refuter pack│ │ ops pack │
│ onboarder │ │ second read │ │ (the viewer)│
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
└────────────────┼────────────────┘
▼ every model call
┌───────────── gateway + Spotlight ─────────────┐
│ verifies each turn's warrant and budget, │
│ logs tokens, cost and errors per session │
└───────────────────────────────────────────────┘
▲
│ register, deploy, start · hold · stop runs,
│ read status and logs (CLI lane, ssh)
Claude CodeEvery instance polls the control plane for its lease: which pack version it may run, its caps, and whether it is enabled. The gateway checks a short-lived warrant on every turn and refuses calls once a deployment’s budget is spent. A person can stop a deployment from the console, and every instance of it stops within one poll, on every machine.
AgentHippo is built to be driven by agents as well as people. The control plane publishes a skill file and an OpenAPI description, so Claude Code registers agents and downloads their setup bundles itself. The runtime reports why a step stopped as a distinct exit code (model not served, session budget spent, out of credits, admission deferred). That lets Claude Code tell a platform fault from an agent’s mistake and act on it: hold a run before a step, stop a campaign, restart a container, or move a run to a fresh deployment when the old one hits its cap. Raising a cap, rolling back or lifting a stop requires a signed-in person. The agent doing the tuning can stop other agents, but it cannot spend more than a person allowed.
The viewer pack keeps every finished campaign: 236 so far, each with its score and cost.
One run of the multi-agent workflow. The orange dashed lines are feedback: a failed check sends its exact failure back to the agent for another round.
What the loop found
1. The instruction drove the cost more than the number of agents
The AP workflow told each agent to do one exhaustive job by hand: read every document, check every finding, read every value twice. It made 400–600 model calls per drop. The single casebook agent is told what to achieve, not how. It writes its own parser and rules, and its memory holds the reviewer’s verdicts. It made 15–33 calls per drop. Same model (gpt-6.1-sol on Codex), cost per campaign:
| World | Workflow (main) | Workflow + triage | Lean workflow | Casebook agent |
|---|---|---|---|---|
| C (auto dealer, AP) | $1.67 · 29/0/0 | $4.49 · 29/0/0 | $2.49 · 29/0/0 | $1.69–1.82 · 29/0/0 |
| T2 (family law) | $4.77 | $8.75 | $4.40 | $1.33–1.49 · 14/0/0 |
| G (loan underwriting) | $7.93 | $6.86 (incomplete) | $5.10 | $2.04–2.08 · 16 found, 0–1 false |
On G, the share of amounts exact to the cent rose from 0.39 with the workflow to 0.97–1.00 with the casebook. Adding agents to check other agents cost as much as the work itself and repeated the same blind spots.
2. Haiku 5.5 with learning beats Opus 5.5 without it
On world D (the property manager, 28 findings), single agents with no memory, one session per drop:
| Agent | Found / false per drop | $ per drop |
|---|---|---|
| Haiku 5.5, no memory | 11–13 found, 58–59 false | — |
| Sonnet 5.5, no memory | 18–22 found, 10–50 false | $1.45–2.54 |
| Opus 5.5, no memory | 16–18 found, 6–46 false | $1.92–2.55 |
| Haiku 5.5 + casebook, after one review | 21–23 found, 0–3 false | ~$0.30 |
| Haiku 5.5 + casebook + AP standard in memory | 28 / 0 / 0 | — |
The first and fourth rows use the same model. In the fourth, the agent’s memory held the reviewer’s verdicts and its own scripts. That took the false findings from 58 to between 0 and 3, at about a seventh of Opus’s cost per drop. With the AP standard also in memory, it found all 28 with nothing false.
3. Learning carries over from one review
Haiku 5.5 on Claude Code, with no AP standard, 18 repeats across six worlds. A person reviewed drop 0 once and gave each case a verdict: upheld, amount corrected, refiled or not a case. Some examples:
| World · repeat | Drop 0, before review | Later drops |
|---|---|---|
| A · r2 (AP) | 24 / 41 / 6 | 29 / 0 / 2 |
| C · r3 (AP) | 7 / 35 / 22 | 24 / 0 / 5, then 27 / 0 / 2 |
| D · r2 (AP) | 18 / 47 / 10 | 23 / 0 / 5 |
| T2 · r3 | 13 / 2 / 1 | 14 / 0 / 0 on every later drop |
The agent learned from the first review in 17 of 18 repeats. In the one failure, on AP world B, the agent read the reviewer’s verdicts and overrode them with its own reading of a price-increase clause. Its check recorded the mismatch but did not act on it. The next change sends a failed check back to the agent before it reports.
4. The engine is a parameter, and it interacts with the model
The same pack, the same model and the same world D, run once on each engine. Cost is at the gateway’s price for each model:
| World D, same model | Engine: Claude Code | Engine: Codex |
|---|---|---|
| Sonnet 5.5 single agent, drop 0 | 18 / 50 / 10 · $1.98 · 41 calls | 20 / 38 / 8 · $1.47 · 36 calls |
| Sonnet 5.5 single agent, drop 1 | 20 / 49 / 8 · $1.83 · 40 calls | 21 / 40 / 7 · $1.45 · 41 calls |
| Haiku 5.5 casebook agent + AP standard, drop 0 | 28 / 0 / 0 · $0.31 · 66 calls, 1 session | 23 / 4 / 5 · $0.68 · 209 calls, 5 sessions |
| Haiku 5.5 casebook agent + AP standard, drop 1 | 28 / 0 / 0 · $0.33 · 84 calls, 1 session | failed: 12 sessions, 593 calls, no findings |
For the Sonnet single agent, Codex came out slightly ahead, with a little more found, fewer false findings and about 20% lower cost. For the Haiku casebook agent, the result reversed. Claude Code finished each drop in one session. Through Codex, Haiku spent most of its output on reasoning and kept exhausting the session’s token allowance. Each new session found nothing on disk to continue from, and drop 1 never finished. That failure led to a change that helps on any engine: the agent now keeps its progress notes current as it works, so a fresh session can pick up where the last one stopped. Each cell is a single run, so the gaps are indicative, not settled. The point for tuning is that the harness belongs in θ, and its best value depends on the model and the job.
Caveats. These are development worlds that have each been run many times; sealed fresh seeds are the real test. The reviewer in these runs is a stand-in that reads the answer key, which makes the learning curves optimistic.
Where researchers and engineers fit
Nobody ran these experiments by hand. We worked in conversation with Claude Code. We asked it to run the workflows on the training worlds and collect the results. It opened the logs for every missed and false finding and explained each one: an invoice read from the wrong line, a rule that fired on a look-alike, a verdict the agent overrode. We discussed with it what to try next. It implemented the change in the packs or the workflow file and started the benchmark again on a pinned commit. The researchers chose the directions and decided which results to keep. Claude Code did the running, the scoring and the first read of the data.
Choosing directions is the most important job in the loop. It resembles a professor working with PhD students: the professor does not run every experiment but decides which questions are worth asking, reads the results critically and recognizes when a promising line is a dead end. Claude Code did the experiments quickly and tirelessly. The judgment about where to go next came from people.
Can the loop close completely, with an agent choosing the directions too? We believe it can, given enough tokens and enough machines to evaluate the proposals. Every step is already an agent’s job except deciding, and deciding is about choosing well among many candidate directions. Today that choice is cheaper to make by a person than to test by brute force. As tokens and evaluation machines get cheaper, it becomes practical to have agents propose many directions, run them all against the answer keys under budget caps, and keep the best.
To try this on your own problem, you need a dataset with answer keys, scorers that never change, and the pieces above. AgentHippo provides the packs, deployments, budget caps and logs. Claude Code or Codex operates the loop.