Editorial illustration for Top AI Models Match Average Human in Complex Strategy Game, Trail Experts
AI Models Match Humans in Strategy Games, Trail Experts
Researchers building the next wave of AI benchmarks have a problem: most tests are too short to catch what actually breaks when a language model has to plan ahead. A chatbot answering one question, or completing one coding task, doesn't reveal much about whether it can hold a strategy together over hundreds of decisions. That's the gap a new benchmark called SpireBench tries to close, using an unlikely proving ground: Slay the Spire, the commercial deck-building game where a single bad card pick in the first few minutes can sink a run an hour later.
The researchers behind SpireBench built a harness that hands AI agents the complete, unmodified game, something they say no prior testing setup has done. Earlier attempts at using Slay the Spire to evaluate models, the paper notes, worked with stripped-down or partial versions, making it hard to compare results or pin down exactly where an agent's reasoning falls apart. SpireBench aims to separate what a model already knows about the game, how well it reasons turn to turn, and how much of its outcome just comes down to the luck of a card draw.
Across 20 large language models (LLMs) at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts. This gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind.
Why this matters
SpireBench gives us something most agentic benchmarks don't: a way to separate early competence from sustained judgment. The detail that matters most here isn't that frontier models match average humans overall, it's where they stop matching them. Winning Act One at expert rates and then fading means these systems can execute short-term plans but struggle to carry a strategy across a full run, where early deck-building choices in Slay the Spire compound over dozens of decisions.
For developers building agents meant to run autonomously for hours or days, that's the finding to sit with. Benchmarks built on short, episodic tasks will keep producing flattering numbers that don't predict behavior on anything resembling a real multi-stage project. Researchers evaluating long-horizon reasoning now have a concrete signal: look for where performance decays, not just where it lands.
And founders pitching "autonomous agents" should ask the uncomfortable question SpireBench raises directly, whether their systems are actually planning ahead or just playing the opening well. Twenty models tested, and none closed the gap with human experts once the run got long.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Cloudflare Clef-omni: 30B-A3B multimodal model for audio, video, image and text
From Archives New Jev Decision Index Ranks Open Reproductions at 64.1 Score
Common Questions Answered
What is SpireBench and why does it address limitations in existing AI benchmarks?
SpireBench is a new benchmark that uses Slay the Spire, a deck-building game, to evaluate AI models' long-horizon strategic planning capabilities. Most existing benchmarks are too short to reveal whether language models can maintain strategy over hundreds of decisions, whereas SpireBench tests this by requiring models to make optimal choices throughout an entire game run.
How do frontier AI models perform compared to humans in SpireBench at the highest difficulty?
Across 20 large language models tested at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player overall, but remain far behind expert players. However, the models' performance gap widens significantly as the run lengthens, winning Act One at expert rates but then falling behind as the game progresses.
What does the performance gap between early-game success and late-game failure reveal about current AI systems?
The fact that frontier models win Act One at expert rates but fade in later acts demonstrates that these systems can execute short-term plans effectively but struggle to carry a cohesive strategy across a full run. This reveals a fundamental weakness in sustained judgment, as early deck-building choices in Slay the Spire compound over dozens of decisions and require long-term strategic thinking that current AI models lack.
Why is Slay the Spire an effective game for testing AI strategic planning abilities?
Slay the Spire is ideal for benchmarking strategic planning because a single bad card pick can undermine an entire run, requiring players to make optimal decisions across hundreds of choices that compound over time. This structure naturally exposes whether AI models can maintain coherent strategy across extended gameplay rather than just handling isolated tasks.
Further Reading
- TextAtari: 100K Frames Game Playing with Language Agents - AlphaXiv
- A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities - arXiv
- DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments - arXiv
- TCG-Bench for Difficulty-Scalable Multilingual LLM Reasoning - ICML
- SokoBench: Evaluating Long-Horizon Planning and Reasoning - arXiv
