Same model, different system

A model scores 13.3%. Change the surrounding software, and it scores 38.3%. The model is still GPT-5.6 Sol.

That is the comparison OpenAI reported in July on the public ARC-AGI-3 puzzle set. Its revised harness retained reasoning between actions and used compaction instead of dropping the oldest history. The combined changes produced the higher score with roughly one-sixth as many output tokens. The score was Relative Human Action Efficiency, not a percentage of coding tasks completed.

A model label had hidden an important part of the result: what the model could remember while it worked. Both settings changed; the experiment does not isolate their individual contributions. Choosing a model and choosing a workflow are different decisions.

The environment changes the task

The same issue reaches ordinary coding work through a less glamorous route: memory limits.

Anthropic held the Claude model, harness and task set constant across six resource configurations on Terminal-Bench 2.0. Strict limits could kill a container during a temporary memory spike. Uncapped resources could also enable heavier strategies. The reported success-rate gap between those extremes was six percentage points. Moderate headroom reduced infrastructure failures without a statistically significant score improvement in that experiment.

A failed dependency install and an incorrect algorithm both leave work unfinished. Stabilizing a container addresses the first failure; allowing larger computations changes what the system can attempt.

Consider a constructed example: an agent converts JSON records into a CSV export. It writes a file, announces success and stops. The file opens. Is the job finished?

Each record contains id and name strings and an integer amount in hundredths of a currency unit. IDs must be unique. Write the header id,name,amount in that column order and sort rows by ID lexicographically. Use UTF-8 without a byte-order mark and LF record endings. Preserve Unicode; quote fields containing commas, quotation marks or newlines, doubling embedded quotation marks. Format amounts with exactly two decimal places. Reject duplicate IDs, missing fields and wrong types. Leave the input unchanged.

A screenshot of the file cannot establish all of that. Freeze fixtures and expected output bytes before the agent runs: ordinary records, a quoted multiline name, a Unicode name, a negative amount and invalid inputs. Check the exit status, output bytes and unchanged input hash. Now “finished” has a meaning independent of the agent's explanation.

Google DeepMind’s AlphaEvolve account reports algorithm improvements in Spanner compaction and cache replacement. It points to a useful question for this exporter: which measurable objective should a valid candidate improve? Correctness is the gate; accepted runs, total cost and repair time are the objectives here. The application reports motivate this measurement design, not a gain estimate for our proposed test.

Make one change, then measure the work

Suppose the agent loses track of earlier failed checks while revising the exporter. A larger model is one option. A smaller system change is to preserve a checkpoint the same model can reread: the input hash, attempted patch, checker outcome and remaining failures.

Here is a proposed two-arm test, not a completed experiment. Arm A receives checker results through normal tool responses. Arm B receives the same responses and a rereadable checkpoint after each check. The checker writes the checkpoint from its observations. Change only that mechanism. Use a fresh conversation and directory for each run against the frozen fixtures; alternate arm order.

Fix the shared limits before running. Record those values, the model endpoint and reasoning setting, environment image, request IDs and checker version. Count checkpoint creation and reading inside Arm B's budget. Keep failed attempts in its cost. Leave unavailable measurements blank.

This is a workflow comparison. OpenAI's evaluation guidance distinguishes controlled comparisons from systems optimized separately for their strongest performance. The claim must match the setup. Changing the model, memory limit and checkpoint together might find a better workflow; it would not isolate the checkpoint's effect.

A checkpoint also records observations, not guarantees. A saved “export complete” message cannot replace reading the resulting file. Persisting a tool result does not establish exactly-once execution or make an ambiguous remote action safe to repeat.

If Arm B accepts more exports within the fixed budget, or reduces repair time without losing correctness, keep the checkpoint. If it only adds tokens and another file to maintain, remove it. If both arms fail the same Unicode case, improve the exporter or its specification before escalating the model.

The useful unit is the system that produces an accepted result. Measure that system, then change the part responsible for the work left unfinished.