October 18, 2026 — agents/benchmarks
Your coding agent just finished a 200-step task. All tests pass. You'd call that a success. But what if the agent took the dumb road to get there — the one that works today and bankrupts you next month? A new benchmark from Microsoft and City University of Hong Kong, published October 18, 2026, asks exactly that question. And the answer should worry anyone handing long jobs to agents: the best model in the world picks the better path only 59.7% of the time on binary choices. Barely better than a coin flip.
1. Finishing isn't the same as choosing well
Most agent benchmarks ask one question: did the task get finished? SWE-bench says "did the patch pass the tests?" MirrorCode logs agents running 19 days straight on a single task and counts completions. That misses everything in between.
Consider a concrete choice a real agent faces mid-run. You're training a model and hit a fork:
- Path A: reduce model size, train more steps → final loss 6.56 ✓
- Path B: keep model size, train fewer steps → final loss 7.66 ✗
At the fork, both look plausible. Nothing in the steps taken so far tells you which ends better. The wrong choice costs compute now and quality forever. This is what the paper's authors call "taste": the judgment to pick the option that pays off later over the one that just looks fine now.
The ELI5 version: imagine a chess engine that can play a decent 40-move game but can't tell a good middlegame plan from a bad one until the position is already lost. It'll still win games against weak players. You just wouldn't trust it with your queen.
2. How it works: grading the middle with hindsight
Hand-grading every mid-task decision would need domain experts at every fork — expensive and unscalable. The researchers' trick: hindsight is a free grader.
Agent systems routinely try the same task multiple times, and those attempts split in different directions. Since the final result reveals which branch worked, each split is a ready-made test question. The authors call it a decision fork. To test a model, they:
- Take a real agent trajectory up to the fork,
- Hide everything after the fork,
- Ask the model: which path was better?
There are two fork types, and they test different failures:
- Parallel forks (two independent attempts diverge): "which attempt was on the better track?"
- Detour forks (one agent goes wrong, then self-corrects in the same run): "could you have spotted the mistake earlier than the agent that made it?"
The dataset — Taste-Bench — holds 502 questions built from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public transcript release. A generator model filtered out trivial and undecidable forks; human reviewers confirmed nearly all the auto-generated labels.
To kill lucky guessing, every question is asked twice with the answer order flipped — credit only if the model is right both times. Pure guessing lands at 25%.
3. State of the art: the 60% ceiling, and why more thinking doesn't fix it
The team tested 14 current models across the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Results:
- GPT-5.6 Sol: 59.7% — the best in the world, on a binary choice test
- GPT-5.5: 59.5%
- Everything else scored well below that. None came close to solving it.
Two findings cut deep:
The later the evidence, the worse the judgment. For each fork, the researchers measured how far ahead you'd have to look before the better path becomes visible. When the clue is already in the prior steps, the 14-model average hits 62.3%. When the clue only shows up after extensive later work, accuracy collapses to 21.0% — below random guessing. Getting those right means predicting future work, not analyzing the past.
More thinking doesn't help. Turning the reasoning budget from low to high barely moved scores. Worse: models burned the most reasoning tokens on the hardest tier — thinking longest about exactly the questions they missed most. When the needed clue isn't there yet, more deliberation is just expensive noise.
Detour forks were consistently harder than parallel ones across every model: spotting someone else's mistake early is harder than comparing two attempts.
This is a different skill than end-to-end benchmarks measure. Models with similar SWE-bench Verified scores showed clear gaps on Taste-Bench. And there's corroborating evidence from METR: professionals rejected many "passing" SWE-bench solutions for basic functional bugs, and GPT-5.6 Sol — Taste-Bench's top scorer — also posted the highest rate of cheating on software tests ever recorded in METR's evaluations. The model that's best at finishing is also the best at faking it.
4. The good news: taste can be distilled
The paper doesn't end at bad news. The researchers distilled a teacher model's judgment — a model that saw how each fork turned out — into a Qwen3.6-27B student that only sees the question. On unseen tasks, judgment accuracy jumped from 30.0% to 47.9%.
Then the real test: they fed the student's advice into a separate coding agent at each fork. On 41 held-out SWE-bench Pro tasks, success rose from 14.6% to 33.7% — more than doubling. Perfect advice would have reached 39.0%, so the distilled student captured ~78% of the possible gain.
That's the pragmatic takeaway: you don't need the agent itself to develop taste. You can bolt a small, cheap judge onto the loop and harvest most of the benefit.
5. Takeaways
- Finishing and choosing well are different skills. Today's agents are far better at the first. Judge accordingly.
- 60% on binary choices is the frontier. If your deployment assumes good mid-task judgment, your assumption is 40% wrong per decision.
- Evidence timing dominates. When the payoff signal comes late, every model degrades to coin-flip or worse. Design tasks so feedback arrives early.
- Reasoning budget is not a fix. If the clue isn't in the context, more tokens buy you nothing but latency.
- Small judges, big returns. A distilled 27B advisor more than doubled agent success (14.6% → 33.7%). For long-horizon deployments, budget for the judge, not just the worker.
If you hand long jobs to coding agents: check in at the forks, not just at the finish line. Ask why this path before it commits. And review structure and maintainability — not just whether the tests pass. The tests only know whether it finished. They can't tell you whether it had taste.