The Coding-Agent Trial: Shortlist to Adoption Verdict on Your Repo

You are choosing which AI coding agent, if any, your team should adopt, and you have decided not to trust a public leaderboard to make the call. This guide is the end-to-end trial procedure for engineering managers, technical founders, developer-relations leads, and technical sourcers: what to measure before the trial starts, how to isolate harness effects from model effects, the sample size that holds up, and the numeric thresholds that separate adopt from reject. Follow it start to finish and you will produce a defensible adopt, hold, or reject verdict scored against your own repository, not against benchmark scores.

Why a leaderboard cannot make this decision for you

A public benchmark score tells you how an agent performed on old, public, contaminated tasks under someone else's scaffold. It does not predict how it behaves on your private code, your review process, or your definition of done. Two mechanisms break the leaderboard-to-adoption inference, and both are measurable.

The first is contamination. In SWE-bench, 32.67% of successful patches involve direct solution leakage from the issue text, and over 94% of Verified issues and their ground-truth pull requests predate the knowledge cutoffs of leading models. The agent may be recalling the answer, not solving it. The second is that the score is not even the score you think it is: 31.08% of passing SWE-bench patches ride inadequate oracles, and OpenAI's manual audit of 138 o3 failures found 59.4% were caused by test flaws, which led them to recommend discontinuing SWE-bench Verified reporting entirely.

So the question "how to evaluate AI coding agents" has one honest answer: run a coding agent trial on your own codebase. Everyone says this. Almost no one tells you the baselines to capture, the task set to draw, the sample size, or the decision rule. That gap is what this guide fills.

The baselines to capture before the trial starts

Capture your delivery baselines for human-authored changes first, because without them any post-trial number is unanchored. The load-bearing frame is the DORA delivery metrics plus a rework signal, pulled from your version control and CI over roughly two weeks of existing data.

The DORA research program defines change lead time, deployment frequency, mean time to restore (MTTR), and change failure rate (CFR), which is the percentage of deployments to production that result in a failure. It adds a fifth metric, rework rate, which measures the amount of code rewritten or deleted shortly after being committed. That fifth metric is the one that catches AI-specific damage.

Resist the temptation to lean on suggestion acceptance rate. The number of generated lines and raw acceptance rates say little about whether throughput, quality, and stability actually improve. Acceptance is an activity metric; you need outcome metrics.

Note "queue depth" is not an established public baseline, so capture review-queue and work-in-progress numbers from your own tracker rather than citing a standard figure. The 2025 DORA report, drawing on nearly 5,000 professionals, found that teams increasing AI adoption experienced worsened change failure rate and deployment rework rate even as individual output increased. Your baseline table is what lets you see that inversion when it happens.

The trial size that actually holds up

Size the trial off the strongest published template and adjust down deliberately. METR ran a randomized controlled trial to measure how early-2025 AI tools affect experienced open-source developers working on their own repositories, using 16 experienced developers, 246 tasks, and mature repositories, with tasks averaging two hours each. That is the shape to imitate: real developers, real repos, tasks drawn from work that would normally happen anyway.

Most teams cannot muster 246 tasks. A defensible compromise is 30 to 50 real issues per repo. The harder sizing question is runs-per-task, because agents are non-deterministic. Setting temperature to zero does not guarantee identical results; accuracy can still vary by 15% across runs. Worse, τ-bench found agents achieving 60% pass@1 may exhibit only 25% consistency across multiple trials. One lucky pass is not capability.

Running an agent 10 to 20 times per task gives proper statistical coverage but is expensive. The defensible floor is k=3 with a pessimistic bound: use the worst of the three runs, not the best, as the task's score. Spend your 10-to-20-run budget only on the highest-risk tasks where a flaky pass would move the verdict. Choose k before you run; deciding after you see results is how motivated reasoning enters.

The task set: what to draw and how to judge it

Seed a private task set with the categories that mirror real work and stress different agent skills, then judge against tests while treating those tests as fallible. METR's issues were bug fixes, features, and refactors that would normally be part of regular work. For a trial, draw bug fixes, test generation, dependency upgrades, and small refactors, because they exercise reading, writing, and integration in different proportions.

Each task needs clear acceptance criteria and a passing or failing test attached. But a passing test is not a passing verdict. About 31% of instances with passing patches rely on insufficiently robust test suites, allowing plausible but semantically wrong patches to pass. For high-risk tasks, manually review passing patches rather than trusting the green check.

Judge stability post-merge, not just at merge. In AI-augmented workflows a low CFR can hide a high rework rate: if a team is fixing AI-generated errors before they reach production, they avoid a deployment failure but lose developer effectiveness. Track rework and churn on agent-authored changes across the trial window. GitClear research shows churn rising from 3.3% to 5.7-7.1% alongside AI adoption, and Faros AI found bug rate rising 9% per developer. If your rework rate climbs like that, a clean pass rate is misleading you.

A green test suite is a claim, not a verdict; on high-risk tasks, read the patch yourself.

Isolating the harness from the model

Before you credit any score to the model you are considering, run an ablation, because the harness moves scores as much as or more than the model does. The harness, also called the scaffold, is the wrapper that controls prompts, tools, retries, and context management around the raw model. Same model, different scaffolding can vary by 10 to 20 percentage points on the same benchmark.

The evidence is stark. On the Morph leaderboard, six leading frontier models span only 4.9 percentage points under the standardized SEAL scaffold, from 41.0% to 45.9%. Hold Claude Opus 4.5 fixed and vary only the harness, and that range widens to 9.5 points. HAL reports double-digit cross-scaffold gaps of 34 points for Claude Sonnet 4.5, 34 for GPT-5 Medium, and nearly 48 for o4-mini, far larger than the cross-model spread under either scaffold. Independent monitoring reports 11 to 15 points of scaffold-only variation on SWE-bench Verified.

The practical consequence: when you compare coding agents, a harness change on your own repo can dwarf the difference between two models. The scaffold taxonomy behind this ranges from tool counts of 0 to 37, through seven context-compaction strategies, to five loop primitives across 13 systems. So run the same private tasks twice: once holding the model fixed and swapping the harness, once holding the harness fixed and swapping the model. Produce a delta table that attributes points to each axis. Without it, you will buy a model when you actually needed a better wrapper, or reject a strong model trapped in a weak one.

Finding the right internal or external reviewer for an ablation is its own sourcing problem, and describing the person in plain English is faster than reconstructing it from filters. Refolk turns a description like that into a ranked list of engineers who actually own these dashboards, which removes the friction of guessing who to ask.

The end-to-end trial procedure

Run these eight steps in order. Owners and rough durations are noted so you can staff and schedule the trial rather than improvise it.

Note one deliberate disagreement to resolve up front. Some practitioners run 10 to 20 times per task for full statistical coverage; others set k=3 as a floor and spend the extra runs only on high-risk tasks. Pick before you run, and write down why.

The decision rule: adopt, hold, or reject

Turn the deltas into a verdict by comparing them against your baselines and requiring the difference to clear run-to-run noise. A pass-rate improvement smaller than your measured variance across runs is not a signal; it is noise wearing a suit. Recall that accuracy can vary 15% across runs even at temperature zero, so your significance bar has to sit above that.

Read three axes together, not one. Throughput, quality, and time, because they move independently and sometimes in opposite directions. The mandate study of 802 developers and 196,212 pull requests reached 2.09x baseline throughput, with a within-developer gain of 1.5x rising to 2x by nine months on tool. Against that, METR's mature-repo developers were 19% slower. Maturity, not the tool, drives the sign, so weight your verdict by how expert your team is on the target repo.

A workable rule of thumb, calibrated to the dossier's evidence and adjustable to your risk tolerance:

• Adopt when the agent shows a throughput or lead-time gain that clears run-to-run noise, and rework rate and CFR do not worsen against baseline, and the ablation attributes the win to something you control.
• Hold when throughput improves but rework or CFR degrades, or when the win is real but attributable mostly to a harness you have not committed to. Re-run with the harness fixed before deciding.
• Reject when the agent fails to clear noise on throughput, or degrades stability without a compensating gain, or its passes survive only weak oracles under manual review.

Remember the maturity caveat when rejecting. A pilot on expert-owned code can show a slowdown and be rejected prematurely, when gains would have grown to 2x by month nine on less familiar work. If your trial repo is the one your team knows best, treat a modest slowdown as expected rather than disqualifying.

How this trial goes wrong

Most failed trials fail the same eight ways, and each has a false positive and a specific check. This is the section to reread before you sign the verdict.

1. Trusting acceptance rate as a quality signal. High acceptance can sit on top of quiet rework. Check by pairing acceptance with rework rate, because a low CFR can hide a high rework rate.
2. Single-run scoring. One lucky pass reads as capability. Check by running k greater than or equal to 3; a 60% pass@1 agent may show only 25% consistency across trials.
3. Weak test oracles. A patch "passes" but is semantically wrong, as in 31.08% of cases. Check by manually reviewing passing patches on high-risk tasks.
4. Contaminated task set. The agent recalls the answer from training. Check by using recent, private, or post-cutoff tasks; 94%-plus of old benchmark issues predate model cutoffs.
5. Attributing a harness win to the model. The number alone cannot tell you whether the model got better or the harness did. Check by running the fixed-model and fixed-harness ablation.
6. Flat CFR read as healthy. More deploys at the same rate means more absolute incidents. Check by reading rework rate alongside CFR.
7. Expecting instant gains on mature repos. A pilot on expert-owned code shows a slowdown and gets rejected early. Check against the maturity effect: METR's 19% slowdown versus the mandate study's climb to 2x by month nine.
8. Silent provider model updates. Your pinned assumptions quietly drift. Check by pinning model and judge versions and alerting on any version change.

Keeping the verdict current

A verdict is valid only against the manifest it was measured on, so treat it as perishable and re-test on defined triggers rather than a comfortable calendar. Pin the exact model version used for both your agent and your judge, so when a provider updates silently your eval detects the change rather than absorbing it. A fixed numeric cadence is not publicly established; the honest triggers are model or harness version change plus the contamination-age of your task set.

Two contamination signals should force a private re-test. Models recall file paths from training data up to 76% of the time, and 32.67% of successful patches have involved solution leakage. As your task issues age past model cutoffs, refresh them with new private work so the agent is solving, not remembering.

Adoption itself is concentrated and market-skewed, which shapes who you can staff a trial with and what your team already knows. In Refolk's index, 1,081 US software and senior software engineers list GitHub Copilot against 148 in the UK, and 117 US engineers list Cursor at employers including Uber, Microsoft, and Chewy.

That skew matters because "what the team already knows" is part of your baseline: a team fluent in one agent will show different integration overhead than one meeting it cold. When you need to recruit trial participants, benchmark reviewers, or a platform engineer who owns rework-rate dashboards, describing the person in plain English and letting Refolk return a ranked list is faster than reconstructing the query from filters. Re-run the trial when a trigger fires, refresh the task set as it ages, and keep the manifest under version control so the next verdict is comparable to this one.