What it does
Two agents — an optimist and a pessimist — build the strongest case they can for and against a decision, from the same evidence, without ever seeing each other's output. A mediator, on a different model family than the debaters, reads both and does one job: find the crux — the single unknown that, if resolved, would end the disagreement — and decide whether a verdict is even warranted yet.
Its opposite number is AgentDesk — same engineering discipline, aimed at the opposite kind of problem.
Why this isn't AgentDesk
AgentDesk is a linear pipeline for tasks where a ground truth exists — a claim in an outreach email either traces to a source URL or it doesn't, and the whole system is built around mechanically checking that trace. Crux is for the other kind of question: build our own system or buy one, take the offer or walk, sign the contract or don't. There's nothing to verify — only competing reasoning, and usually one fact nobody has looked up yet that both arguments quietly depend on.
Same spine, opposite shape. AgentDesk gates — pass or block, with a retry budget that runs out and escalates to a human. Crux never gates; it produces a decision aid, and it gets exactly one pass, no retries, because sending a weak position back for a stronger case would mean the system is coaching the debate rather than judging it. Where AgentDesk's hard check asks whether an email still reads as sendable with the company name redacted, Crux's asks whether one side argued more than 25% longer than the other — length reading as conviction is the bias a synthesizer falls into first.
How it's wired
- Parallel isolationThe optimist and pessimist get the byte-identical evidence pack and never see each other's output. A position that references an opponent it was never shown is rejected outright — the phantom-opponent check.
- Symmetry enforcementBoth sides get the identical word budget, recomputed in code and never trusted from the model's own count, plus a 1.25× length-ratio check that fires even when both individually fit the budget.
- Order randomizationWhich position is presented to the mediator first is randomized and the seed is logged, so any reading-order bias is auditable after the fact.
- ContractsEvery handoff — evidence pack, position, crux report — is a typed JSON schema. The mediator has no search tools and works only from the two positions and the pack.
- GateThe mediator must name a concrete
primary_cruxbefore any verdict, checked in code. A vague crux attached to a confident verdict is the averaging behavior this system exists to block. - RecordEvery run's seed, presented-first order, and outcome is written to a run log.
Measured results
evals/run_eval.py scores 15 cases, each a named mutation of one baseline debate:
8 of 12 bad debates blocked before spending a single mediator call, at zero false
positives. The four that escape the deterministic layer need real judgment, not arithmetic:
- A phantom opponent in substance, not phrasing. One position rebuts the other's actual argument without using any flagged phrase — code checks for phrases; it can't see that a position is answering the other.
- A generic position. Right length, valid references, no phantom-opponent phrase, reads well — and swap the subject for any other build-vs-buy call and it still sends. Code can't catch contentlessness.
- A mislabeled inference. An extrapolation marked as a direct read with high confidence. Code can't tell a direct read from a stretch — that's the mediator's job.
- Evidence drift. A claim cites a real source, then restates it as something stronger than the source actually said.
run_eval.py --live would add real mediator calls and measure whether the model
catches those four. It hasn't been run — this project makes no API calls yet.
Known limits
- The phantom-opponent check is a phrase denylist, and denylists leak — a position can rebut a view it never saw using none of the patterns on the list. This is the same mechanism as AgentDesk's banned-phrase list, and it was caught leaking the same way: on a paraphrase neither list anticipated.
inference_typeandconfidenceare both self-reported by the debaters. Nothing in the deterministic layer validates either label — only the mediator's own judgment can catch a stretch dressed as a direct read.- Single pass means a weak position produces a weak report. There's no retry to send an underperforming debater back for a stronger case — a deliberate contrast with AgentDesk's retry budget, not an oversight.
- The gate that checks the mediator's crux report verifies shape, not correctness. It can confirm the named crux is specific and non-vague; it can't confirm it's actually the load-bearing disagreement rather than a real-but-secondary one.
- The eval set is synthetic — fifteen mutations of one baseline debate. It measures whether checks fire on known failure shapes, not whether the crux the mediator names is the one a domain expert would name.
- No live run has ever happened. Every terminal state, every scenario, and the entire eval above are demonstrated on fixtures — real code paths, pre-written JSON, no API calls made.
What I'd do next
- Run
evals/run_eval.py --liveonce there's API budget for it — the four escapes above are the interesting question and it's still unanswered. - Test whether a domain expert would actually name the same crux the mediator does. Nothing in this repo measures that yet.