Aviral GargCompare AI systems by the work you can accept and the resources needed to get there. A low price for one response tells you little about the cost of finishing a task that needs corrections, tool calls or several attempts.
Define an accepted completion
Choose a concrete result before running a comparison. For a coding task, that might mean a patch passes the agreed checks and a reviewer accepts the change. A confident explanation alone does not establish that result.
Anthropic’s evaluation guide distinguishes a task, an attempt and the resulting environment state. It also separates the agent’s recorded actions from the outcome. This helps explain why “the agent says it succeeded” is a different observation from “the requested change exists.”
Hold the comparison steady
Use the same task set, acceptance rule, tool access, harness version, time budget and retry policy. A harness is the setup that runs and checks the work. Record the exact model version and configuration too.
SWE-bench’s evaluation guide offers a concrete example: it applies generated patches and runs repository tests in Docker. Its result categories distinguish an evaluation producing a report from a patch resolving the issue. Those counts answer different questions.
An original hypothetical example
Imagine two unnamed options attempting the same ten tasks under the same allowed budget. The figures below are invented arithmetic, not benchmark results, invoices or provider prices.
| Option | Total attempt cost (USD) | Accepted tasks out of 10 | Cost per accepted task |
|---|---|---|---|
| A | $10 | 5 | $2.00 |
| B | $12 | 8 | $1.50 |
For this example, use:
Cost per accepted completion = total cost of all attempts ÷ accepted completions.
Include failed attempts and retries in the numerator. Count each accepted task once. With zero accepted tasks, report that outcome and the spending; the ratio is undefined.
Option B has the lower ratio in this toy example even though its total spending is higher. Ten fictional tasks establish no model ranking or prediction about your workload.
Keep the missing details visible
Report elapsed time, retry counts and human review time separately. Specify which model, tool or other charges the total covers. Usage multiplied by rates is an estimate unless reconciled with attributable billing records. Missing cost coverage should stay visible.
Learning question: What happens if you exclude failed attempts?
Answer: You undercount the resources spent reaching accepted results, making the comparison look cheaper than the complete workflow.