GPT-5.6 Sol Scored 7.8% on ARC-AGI-3. The Problem Was the Harness, Not the Model.

When we first pulled up the ARC-AGI-3 leaderboard, our immediate reaction was confusion.

GPT-5.6 Sol had already solved the cycle double cover conjecture, a longstanding open problem in mathematics. It beat Pokémon FireRed using a vision-only setup. Yet on ARC-AGI-3, a benchmark built from 2D puzzle games, it managed a mere 7.8%. GPT-5.5’s performance was even more extreme, scoring 0.4%, which essentially means it failed to play the games at all.

How does a model capable of beating Pokémon get stuck on simple 2D puzzles?

The answer had nothing to do with the model’s underlying capabilities. The problem was buried in how the benchmark was being run. Two API settings were turned off: retained reasoning and compaction. We flipped those switches on, and GPT-5.6 Sol’s score on the public task set jumped from 13.3% to 38.3%. At the same time, its output token count dropped to a sixth of what it originally was.

The technical mechanism behind this is straightforward, but the broader implication is easy to miss. Benchmark scores rarely measure a model in isolation. They measure a bundle of decisions: how the API is configured, how the harness manages memory, and how prompts are passed through. You can take the exact same model, change the test conditions, and see the score shift by a factor of three.

What Exactly Does ARC-AGI-3 Measure?

ARC-AGI-3 tests how well AI agents learn and reason by having them explore unfamiliar 2D games without any explicit instructions. The agent has to infer the rules from feedback and figure out how to advance through the levels. You can play 25 of these demo games yourself right now on the ARC website.

The benchmark intentionally uses a highly generic harness. There are no tool calls, no special features, and no model-specific accommodations. ARC’s logic for this is sound on the surface: a stripped-down harness makes model shortcomings more visible, and it theoretically creates a level playing field for comparing different models. Commercial development doesn’t work this way. Product teams build custom harnesses for each model, tuning the setup to play to the model’s specific strengths and compensate for its quirks.

Looking at GPT-5.6 Sol’s track record in gaming, the ARC-AGI-3 results felt like an outlier. It cleared Pokémon FireRed with a vision-only harness. It beat Slay the Spire using Codex computer use. It pushed through the early stages of Baba Is You. What made ARC-AGI-3 so different?

The benchmark wasn’t testing a different model. It was testing the same model under a completely different set of memory constraints than the ones it was trained and deployed to use.

Summary: ARC-AGI-3 uses a minimal harness to test online learning and reasoning, but this pursuit of “fairness” inadvertently forces models to operate without the memory management they rely on in production.

Why Did the Model Look “Dumb”?

We followed the trail blazed by ARC’s own analysis of GPT-5.5’s failures and went digging into GPT-5.6 Sol’s attempt logs. The behavior matched ARC’s description perfectly. The model lingered on every single action, struggled to make forward progress, and generally looked like it couldn’t grasp what was happening on screen.

But looking a layer deeper revealed a different story. The confusion was not inherent to the model. It was a direct result of the harness settings.

The first issue we noticed was that every single private reasoning message was being discarded after a game action completed.

Our models are trained to generate private reasoning messages before they output a final reply or a tool call. This is the model’s internal workspace where it analyzes the current state, recalls previous attempts, and plans its next move. In ChatGPT and Codex, these messages are retained as part of the conversation history. The ARC-AGI-3 harness deleted them the moment the action was submitted.

The practical effect of this was that GPT-5.6 Sol had to figure out the game from scratch on every single turn. It could still see a log of its past moves and some brief notes attached to them. But the actual thought process behind those moves—the strategies, the insights, the rejected hypotheses—was gone. Imagine trying to solve a complex math problem, but after every calculation step, someone tears up your scratch paper. You only get to keep the final number. Trying to solve the next step becomes nearly impossible.

The second issue was the harness’s approach to context limits: rolling truncation.

When the conversation context exceeded 175,000 characters, the oldest messages were simply dropped. This meant GPT-5.6 Sol wasn’t just losing its thoughts. It was losing its actions. Early observations, dead ends it had already ruled out, and foundational discoveries about the game mechanics were silently deleted as the game progressed.

These two mechanisms working together guaranteed that the model could never accumulate any effective learning. It was not “learning to play a game.” It was repeatedly being introduced to the same game for the first time.

Summary: The official harness discarded reasoning messages after every move and used rolling truncation to delete old actions, forcing the model to operate in a state of perpetual amnesia.

What Happens When You Keep the Reasoning?

We rebuilt the ARC-AGI-3 harness using our Responses API. The API handles context management in a way that matches our production environments. For GPT-5.6, you simply pass the previous response ID, and the API automatically retains the reasoning messages across tool calls and conversation turns. There is no manual extraction or stitching required.

Flipping on retained reasoning produced two immediate, visible changes.

The first was a drop in per-action thinking time. The model no longer needed to re-interpret the entire game state from scratch before every move. It had already done that work. It could pick up its internal train of thought and move forward.

The second change was more significant. The model started demonstrating actual, cross-step learning. It remembered what it had tried. It recalled which approaches failed and which showed promise. It adjusted its strategy based on accumulated evidence. This isn’t some exotic meta-cognitive skill. It is the most basic form of “remembering what you did.”

I initially assumed retaining reasoning would mostly be an efficiency win—saving a few reasoning rounds and cutting down on token waste. The actual results proved me wrong. The impact on final accuracy was far larger than the impact on token cost. When a model can remember its reasoning, it builds a genuine understanding of the game rules during early trial-and-error phases. It stops spinning its wheels. In longer levels, this effect compounds. The further you get into a game, the wider the gap becomes between a model that remembers and a model that doesn’t.

Summary: Retaining reasoning eliminates the need to re-learn the game state each turn, reducing thinking time while significantly improving strategic coherence over multiple steps.

Why Compaction Beats Rolling Truncation

The second modification was replacing rolling truncation with compaction.

Rolling truncation is a blunt instrument. When context gets too long, you chop off the oldest messages. This creates two distinct problems. First, the model loses early observations that might still be highly relevant in the late game. A rule discovered in level one might still apply in level five. Second, the model ends up spending the vast majority of the task operating with a nearly full context window. Running near the context limit subtly degrades performance because the model’s attention is spread thinner across a denser block of text.

Compaction handles long contexts differently. Instead of deleting history, the Responses API summarizes it. The model’s hard-won understanding of the game rules and its validated strategies are not thrown away. They are preserved in a compressed format that still carries the essential information.

When we enabled compaction on ARC-AGI-3, GPT-5.6 Sol maintained its understanding of each game across longer runs and achieved higher scores while generating fewer output tokens.

A quick technical note on the implementation: the official ARC-AGI-3 harness triggers truncation at 175,000 characters. Our implementation uses a 175,000 token limit. In practice, the difference is negligible. The overwhelming majority of the text in these game logs consists of action grids, and our tokenizer processes those grids at roughly a 1:1 character-to-token ratio.

Summary: Compaction summarizes old context instead of deleting it, preserving early discoveries and preventing the performance degradation caused by operating near a full context window.

The Raw Numbers: How Much Did the Scores Actually Change?

Here is the full data set for GPT-5.6 Sol on the ARC-AGI-3 public task set. The scoring metric is RHAE (Relative Human Action Efficiency), which compares model performance against a human baseline. Based on official gameplay logs, the average human tester scores around 48%. The models are not told how they are being scored, and they cannot see their scores during the game. After each action, they only receive a text representation of the current frame and their current level number.

At max reasoning effort, the score goes from 13.3% to 38.3%. That is roughly a threefold increase. The output tokens per game drop from about 2.9 million to about 485,000—a sixth of the original cost. Seeing the score triple while the cost drops by 83% makes the point clearly.

But the row that actually catches my attention is not the Max tier. It is the Xhigh tier. The official harness yields 7.2%. The new harness yields 25.7%. That is a 3.5x difference. The High tier shows a similar pattern: 5.2% jumping to 13.4%. Retained reasoning and compaction are not marginal tweaks that only matter at extreme parameter settings. They create order-of-magnitude differences in the mid-to-high reasoning ranges.

Think about the reverse scenario. If someone ran a “medium reasoning effort” test using the official harness, they would get a score of 1.5%. They might write a blog post stating that “GPT-5.6 Sol is basically non-functional on ARC-AGI-3.” Technically, that conclusion would be wrong. The model didn’t change. The test conditions did. But readers do not look at harness source code. They remember the number.

Summary: At max effort, scores nearly triple while token usage drops to one-sixth. The performance gaps in the Xhigh and High tiers are even more disproportionate, proving the harness impact is structural, not marginal.

A Side-by-Side Look at a Specific Level

One public task makes the difference painfully obvious. On this specific set of levels, no frontier model on the public leaderboard solves anything beyond the first level. Using the harness with retained reasoning and compaction, GPT-5.6 Sol at max effort clears all six levels.

A sped-up video of GPT-5.6 Sol solving puzzles. The left side uses the official harness. The right side uses the Responses API harness with retained reasoning and compaction enabled.

Watch the two sides of the video. Both are running the exact same model at the exact same reasoning effort. The right side moves noticeably faster. The pauses between actions are shorter. The path through the level is coherent, with very little backtracking. The left side repeatedly loops through areas it has already explored. It revisits dead ends because it literally does not remember being there.

The two central columns visualize how the 175K context window is utilized by each harness. With better memory, the model thinks less per action and progresses much faster.

There is also a visualization of the 175K context window. It shows how space is consumed differently under the two regimes. Under the new harness, the window is used efficiently. Because the model doesn’t need to generate redundant “re-understanding” reasoning every turn, and because early insights are compressed rather than deleted, there is more room for genuine new thinking. Under the official harness, massive chunks of the window are clogged with repetitive, from-scratch reasoning loops. Actual incremental learning gets squeezed out.

Summary: On a public task where no frontier model passes level one on the leaderboard, the new harness clears all six levels by eliminating redundant exploration and making far better use of the context window.

What Should You Do If You Run Benchmarks via API?

This case study is a warning for anyone evaluating models through API calls. You are not just testing the model. You are testing your harness. And your harness is likely having a much larger impact on the final number than you think.

If you are an API developer trying to get an accurate read on a model’s actual performance ceiling, stop writing custom memory management logic. Use the same settings the model uses in production:

• Pass the previous response ID in the Responses API so reasoning messages are automatically retained across tool calls and turns.
• Enable compaction instead of relying on rolling truncation or manual history deletion.

These are not advanced optimization techniques that require delicate tuning. They are the baseline conditions our models are trained under and deployed with. Testing a model with a completely different memory management strategy means testing it in an environment it was never designed to handle. Getting a low score under those conditions is the expected outcome. Getting a high score would be the surprise.

If you are in the business of comparing models, only put weight on evals that explicitly use these settings. If an eval runner uses a generic harness that silently strips away reasoning messages, you are not comparing model capability. You are comparing how badly each model handles having its memory artificially disabled.

This is not the first time we have been surprised by a low score on a public benchmark, only to trace it back to a generic harness dropping reasoning messages. It happens with a frustrating regularity. To their credit, ARC did something valuable here. Their detailed analysis of GPT-5.5’s failures is what prompted us to stop accepting the number and start digging into the harness code.

Summary: Align your test harness with the model’s production environment by retaining reasoning and enabling compaction. Evaluations that strip these features will systematically underestimate model capability and produce invalid comparisons.

Actionable Checklist

• ARC-AGI-3’s official harness deletes private reasoning messages after every single action.
• The official harness uses rolling truncation at 175,000 characters, permanently deleting the oldest messages.
• Rebuilding the harness with the Responses API and passing the previous response ID automatically retains reasoning.
• Replacing rolling truncation with compaction preserves early learning as summaries instead of deleting it outright.
• GPT-5.6 Sol at Max effort: score moved from 13.3% to 38.3%, output tokens dropped from 2,900,997 to 485,485.
• The human baseline for RHAE scoring is approximately 48%. Models do not know the scoring rules or see their scores.
• When comparing models, prioritize evaluation results that explicitly retain reasoning and use compaction.

One-Page Overview

GPT-5.6 Sol initially scored just 7.8% on ARC-AGI-3 not because of a lack of capability, but because the official benchmark harness discarded the model’s private reasoning after every move and used rolling truncation to delete old actions. This forced the model to operate in a state of constant amnesia. By rebuilding the harness with the Responses API to enable retained reasoning and compaction, the score on the public task set jumped from 13.3% to 38.3% while output token usage dropped to one-sixth of its original level. Benchmark scores reflect the combined performance of the model and the harness, and memory management strategies can alter results by orders of magnitude.

FAQ

What is RHAE scoring and how does it differ from a simple pass rate?
RHAE stands for Relative Human Action Efficiency. It compares the model’s performance directly against a human baseline rather than just measuring whether a level was passed or failed. The estimated average human tester scores around 48% on these tasks.

How do you actually enable retained reasoning?
When using the Responses API, you pass the response_id from the previous turn into your next request. The API handles the rest, automatically keeping the reasoning messages attached to the conversation history without any manual parsing.

What is the functional difference between compaction and rolling truncation?
Rolling truncation permanently deletes the oldest messages in the context window when a length limit is hit. Compaction summarizes the existing history into a shorter format, preserving the key learnings and rules the model has already discovered.

Why would a benchmark intentionally drop reasoning messages?
The goal is to create a maximally generic, model-agnostic testing environment to ensure fair comparisons. The tradeoff is that this generic environment diverges significantly from how these models are actually trained and deployed in production.

Does the performance gap still exist at low reasoning effort?
Yes, though the absolute difference is smaller. At Low effort, the official harness scored 0.9% compared to 3.7% with the new harness. The gap becomes dramatically wider at higher effort levels, peaking at the Xhigh tier where the new harness scores 25.7% versus the official 7.2%.

What happened to GPT-5.5 under the new harness?
The source data only provides GPT-5.5’s score of 0.4% on the official harness. Its performance under retained reasoning and compaction was not included in the test results.

Are these findings only applicable to OpenAI models?
The data specifically covers GPT-5.6 Sol and GPT-5.5. However, the underlying principle—that harness memory management drastically alters benchmark scores—applies broadly to any model that relies on chain-of-thought or private reasoning.

How different are 175,000 characters and 175,000 tokens in this context?
The difference is minimal. The context in ARC-AGI-3 is dominated by action grid text, which the OpenAI tokenizer maps at roughly a 1:1 character-to-token ratio.