Editorial illustration for New Research Shows Why Agent Rankings Change After Accounting for Competition
Agent Rankings Shift When Accounting for Competition
New Research Shows Why Agent Rankings Change After Accounting for Competition
A team testing agent configurations ran into a familiar problem: change the model, rewrite the prompt, swap a retrieval tool, and the average score barely moves. One version edges out another by two points. A judge picks a different winner on Tuesday than it did on Monday.
The instinct is to ship whichever config posted the highest average and call it done. That instinct, according to the research, gets the decision backward.
The issue isn't noise in the scoring. It's the frame. Agent performance doesn't break down neatly into better model plus better prompt plus better tool, because those pieces interact.
A prompt that boosts a smaller model can slow down a stronger one. A semantic search tool might pair well with an open-ended prompt and fall apart against a rigid, step-by-step one. Averaging scores in isolation erases exactly the interactions that determine which configuration actually wins in practice.
The alternative starts with a simple constraint: stop scoring answers one at a time, and start forcing direct comparisons between configurations on the same examples, asking a judge to pick only the best and worst.
The γ (gamma) terms shows the how interactions between choices, like the way the model and tool work together. The rankings shift once competition quality is accounted for. Config 1 drops from the raw-win-rate second place to third.
Config 4 jumps from raw-win-rate fourth to second. Config 5 is the cautionary case: it has the strongest β-only baseline on paper, but its total score falls below zero once interaction penalties are included. That is exactly the kind of signal average scoring tends to bury.
The question is not only "which components are strongest?" It is "which config wins against serious alternatives?" Operational Decisions Deploy: Config 7 Config 7 -- GPT-5.4-mini / Contextual Leaper / semantic_search · Total: +0.431 Config 7 is the clear winner at +0.431. Its individual component baseline is solid, but not enough to explain the result by itself. What makes it stand out is that the model, prompt, and tool reinforce each other.
Why this matters
If Config 4 can jump from fourth to second once you strip out the noise of who it happened to be judged against, then every leaderboard-style eval we've been trusting is probably telling us less than we think. Average scores flatten out exactly the interaction effects, the gamma terms, that decide whether a prompt-model-tool combination actually holds up outside the batch it was tested in. For teams shipping agents on the strength of a two-point win, that's a real problem: you might be promoting a config that only looked good because of easy competition in the eval set, not because it's structurally better.
The fix isn't complicated, just unfamiliar. Treat agent evaluation like a ranking problem with opponents, not a scoreboard with fixed scores. That means rerunning old comparisons with this lens before trusting any past "winner." For researchers building eval frameworks, this is a case for building in matchup-aware scoring from the start. For founders, it's a reason to ask your team which of your current defaults were chosen on an average that might not survive scrutiny.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Auto-FL-Research Uses Agents to Automate Federated Learning Algorithm Search
From Archives 60% of Experts Say Humanity's Last Exam Is Necessary and Useful
Common Questions Answered
Why do agent configuration rankings change when accounting for competition quality?
Agent rankings shift because raw average scores mask interaction effects between components like models and retrieval tools. The gamma (γ) terms represent how these choices interact with each other, and once these interaction penalties are accounted for, configurations that appeared strong in isolation may perform worse, while others jump significantly in the rankings.
What does the research reveal about relying on average scores to select agent configurations?
The research shows that average scores flatten out the interaction effects that actually determine whether a prompt-model-tool combination will hold up outside the specific batch it was tested on. Shipping an agent based on a two-point win in average score is problematic because that small margin may disappear once competition quality is properly accounted for.
How can a configuration like Config 5 have a strong baseline but still score below zero after interaction penalties?
Config 5 demonstrates that a strong individual component performance (β-only baseline) does not guarantee overall success when interaction penalties are included. Once the gamma terms accounting for how components work together are factored in, the configuration's total score falls below zero, revealing that its components do not complement each other effectively.
What is the implication of this research for leaderboard-style agent evaluations?
If agent configurations can jump multiple positions in rankings once competition noise is stripped out, then current leaderboard-style evaluations are providing incomplete information about which agents will actually perform well in real-world scenarios. This suggests that teams should be skeptical of evaluation methods that don't account for interaction effects between agent components.
