How to Evaluate a Coding Agent: Traces, Tests, Reverts, and Cost per Merged PR

A coding agent can finish with a convincing summary and a green test run. Neither tells you whether it solved the right problem, how much supervision it needed, or whether the change survived production. A leaderboard score is even further from those questions: it measures performance on somebody else's tasks, under somebody else's harness.

The useful question for a team is narrower: does this agent-plus-workflow deliver acceptable changes on our work, at a cost and risk we can sustain? Answer it with four connected layers: traces, verified outcomes, post-merge rework, and fully loaded cost. No one layer is a verdict on its own.

This is a measurement framework, not a report of an experiment we ran. The example numbers below are fictional. The recommendations are derived from primary guidance on agent evaluations from Anthropic , traces and evals in the official OpenAI Docs , and software-delivery metrics from DORA .

First, define what you are evaluating

“Codex versus Claude Code” is not just a model comparison. An agent's outcome depends on its model, harness, instructions, repository context, tools, permissions, runtime, and human approval points. Anthropic explicitly describes an agent evaluation as measuring the model and harness together. If you change both the model and the task setup between runs, you cannot attribute the difference to either one. Anthropic: Demystifying evals for AI agents

Record four separate units before collecting metrics:

Keep an immutable task ID and connect every attempt, trace, PR, review, merge, deployment, and later fix to it. Also record the repository revision, agent configuration, model version, available tools, and whether a human intervened. Without this linkage, “cost per merged PR” becomes an attractive but untrustworthy spreadsheet cell.

Sample work by category: bug fixes, tests, refactors, documentation, and feature changes should not be silently pooled. A hard migration and a one-line typo fix do not represent interchangeable units of productivity. Compare within a category, then show the overall task mix separately.

1. Read traces to explain behavior, not to reward choreography

A trace captures what happened during a run: model and tool calls, handoffs, results, timing, and failures. OpenAI's agent-evals guide recommends starting with traces to debug workflow behavior, then moving to repeatable datasets and eval runs. Anthropic similarly distinguishes the trace from the final state of the environment.

For each attempt, ask:

• Did the agent inspect the relevant code and reproduce the problem before editing?
• Did it run the checks required by the task, and did it react correctly to failures?
• Did it retry the same unproductive action, ignore a tool error, or claim success without evidence?
• Did it request human help at an appropriate point, especially around ambiguous requirements, credentials, destructive commands, or deployment?
• Were tool, sandbox, or permission denials part of the task outcome rather than silently removed from the record?

Capture useful counts—tool calls, retries, elapsed time, test commands, approvals, and human interventions—but inspect representative failed traces before treating a count as a target. A long trace can show waste, or careful investigation of a genuinely hard bug. A short trace can show efficiency, or a skipped verification step. Anthropic warns that rigidly requiring a particular tool-call sequence can reject valid solutions; grade the outcome first and use the path to diagnose it. Anthropic: agent evals

Traces may contain prompts, file contents, command output, and tool arguments. Apply the same retention, access-control, and secret-redaction rules you would use for engineering logs. Also note missing usage data rather than replacing it with zero: OpenAI's Agents API documentation says trace and token records can arrive after a turn and may be incomplete.

2. Test the change against the task, not merely the existing suite

Before the agent runs, write a short acceptance contract: the behavior to add or fix, the behavior that must remain unchanged, and the checks that can verify both. This avoids grading a plausible patch against a moving target.

Use complementary checks:

1. Task-specific tests: Does the previously failing case now pass? For a feature, do newly written acceptance tests capture the requested behavior?
2. Regression tests: Do existing relevant tests still pass? Run the same check set across compared attempts.
3. Static and policy checks: Typecheck, lint, dependency, security, migration, or protected-path rules where applicable.
4. Human review: Does the patch actually satisfy the requirement, preserve design intent, and avoid needless complexity? Record review time and revision requests.

Tests are strong evidence for what they cover, not proof of everything they omit. Anthropic recommends well-specified coding tasks, stable environments, and thorough tests, while also combining code-based, model-based, and human graders where appropriate. Its distinction between a trace and the final environment state is crucial: the agent saying “done” is not an outcome. Anthropic: Evaluating coding agents

For a practical scorecard, report verified task success as tasks satisfying their predeclared acceptance criteria divided by all eligible tasks attempted. Report PR creation and merge yield separately. A passing test suite, an opened PR, and a merged PR are three different milestones.

For ambiguous work, let reviewers mark “unclear specification” rather than force a pass or fail. Improve the task definition before using that case to rank agents. Check whether an agent edited tests to make them pass by weakening assertions, and keep any hidden evaluation checks unavailable to the agent during the run.

3. Follow the PR after merge: reverts, hotfixes, and rework

Merge is a workflow decision, not a warranty. A change can pass CI and review yet trigger a rollback, require a hotfix, or impose substantial cleanup on the next engineer. Define an observation window—say, 30 days after deployment—before comparing outcomes. Do not count a recent merge as “clean” when its window has not elapsed.

Track at least three labels:

• Attributable revert: the merged change was explicitly backed out because of a defect or unacceptable behavior.
• Attributable fix-forward: a later PR repaired a defect introduced by the change without reverting it.
• Unrelated follow-up: requirements changed, or a later improvement was planned independently. Do not charge this to the original agent.

Require a link to the incident, issue, revert commit, or repair PR and a short human attribution note. Partial reverts and bundled releases need manual review. If the agent's change was only one part of a larger deployment, avoid confidently assigning the whole incident to one PR.

Do not call “reverted PRs / merged PRs” a DORA change fail rate. DORA defines change fail rate at the deployment level: the share of deployments requiring immediate intervention, often a rollback or hotfix. It also tracks deployment rework rate. PR-level reverts and fix-forwards are useful local signals, but they have a different denominator and attribution problem. Show both only when you can measure them correctly.

4. Calculate cost per merged PR without hiding failed attempts

Start with a clearly labeled accounting boundary. A practical pilot can use:

Fully loaded cost per merged PR =
  (all agent/model charges for eligible tasks, including failed runs and retries
   + tool, sandbox, CI, and third-party charges
   + human setup, supervision, review, and repair time valued at a stated rate)
  / merged PRs from those tasks

If the denominator is zero, report “no merged PRs,” not an infinite or missing-looking price. Keep the numerator and denominator visible. Report agent-only spend as a separate number if useful, but never label it “total cost.” An agent that uses cheap tokens while requiring hours of review may be expensive in the way the team actually feels.

OpenAI's observability guide says an agent may make several model calls and advises accounting for root and subagent work, retries, tool use, sandbox compute, and third-party charges. It also warns that recorded usage can be missing or change later and is not a final bill. For subscriptions or included credits, disclose the allocation rule instead of pretending each run has a precise API invoice.

For example, imagine a fictional pilot with 20 eligible tasks, 26 total attempts, 12 opened PRs, and 10 merged PRs. Agent/model and tool charges total $240, infrastructure costs $120, and 12 hours of human review and repair at an explicitly assumed $80/hour add $960. The fully loaded cost is ($240 + $120 + $960) / 10 = $132 per merged PR. Agent-only spend would be $24 per merged PR; using only that figure hides most of this example's work. Neither number tells you whether the ten changes were valuable or durable.

If one of those PRs is reverted, the repair effort belongs in the numerator when it falls within the chosen accounting window. Report the timing: a cost figure at merge and a revised figure after the post-merge window answer different questions. Avoid comparing a mature cohort with a fresh one.

A scorecard a team can actually maintain

Put the raw counts next to every percentage. “One revert out of eight mature PRs” is more honest than a polished 12.5% stability score, especially when another two PRs are still inside the observation window. Keep “unknown” distinct from zero. Do not combine task categories into one weighted score unless the weights represent a real business priority agreed in advance.

Run a small pilot without pretending it is a universal benchmark

Pick a modest set of recent, representative tasks, including failures and awkward maintenance work—not only polished demo tickets. Freeze each task's starting repository revision and acceptance criteria. Give each candidate comparable access to documentation, tools, time budget, permissions, and human support. If a tool cannot operate under the same boundary, record that as a workflow difference rather than quietly relaxing the boundary.

Then run the loop:

1. Log the task and configuration before starting; assign a task category and difficulty estimate.
2. Capture every attempt, including retries, aborts, and cases that never produced a PR.
3. Run predeclared tests and review the diff; document ambiguous failures.
4. Record review, revision, and merge decisions with human time spent.
5. Follow merged changes through the same post-deployment window.
6. Read a sample of successful and failed traces, correct faulty labels, and report results by category with raw counts.

If you compare two agents, alternate or randomly assign similar tasks within each category rather than giving one the easy backlog. A small pilot is for finding failure modes and estimating operational fit, not claiming statistical superiority. If the first result is close, gather more tasks before standardizing a tool.

Public benchmarks can still help you understand what a system has been tested on. They cannot tell you whether it works with your branch protections, codebase, reviewers, security rules, or cost model. The operational scorecard closes that gap.

Make a decision, then keep measuring

Adopt an agent workflow for a task category when it repeatedly produces verified, reviewable changes with manageable intervention, acceptable post-merge stability, and an all-in cost that compares favorably with the team's alternative. Narrow it when it succeeds on tests or documentation but creates expensive review or risky production changes. Reject or redesign the workflow when success depends on skipped checks, undisclosed retries, or human repair that the headline “automation” number excludes.

This evaluation is about a system of work, not a personality contest between models. If you are choosing the interface and harness first, see our Claude Code vs. Codex vs. Cursor decision guide. For the permission and deployment side of the scorecard, see our coding-agent security guide. To understand why two agents using similar models can behave differently, read what makes an agent harness reliable.

The shortest useful summary is: trace the attempt, verify the task, watch the merged change, and count the whole cost. Anything less can make an agent look productive while moving the work somewhere harder to see.