Editorial illustration for CreativityBench benchmark introduces 4K‑entity affordance KB to test LLM creativity
CreativityBench benchmark introduces 4K‑entity...
The world isn't made of hammers and nails. It's a jumble of whatever's heavy enough to pound a nail when the real tool is missing. That simple, messy truth of human ingenuity completely baffles our best AI.
A new benchmark called CreativityBench proves it. Built on a knowledge base of 4,000 objects and over 150,000 annotations about what parts of those objects can actually do, the test doesn't ask for a tool's intended use. It demands the model invent a new one from scratch.
The results are a cold splash of water.
As a first step, we introduce CreativityBench, a benchmark for evaluating affordance-based creativity in LLMs. To this end, we build a large-scale affordance knowledge base (KB) with 4K entities and 150K+ affordance annotations, explicitly linking objects, parts, attributes, and actionable uses. Building on this KB, we generate 14K grounded tasks that require identifying non-obvious yet physically plausible solutions under constraints.
Evaluations across 10 state-of-the-art LLMs, including closed and open-source models, show that models can often select a plausible object, but fail to identify the correct parts, their affordances, and the underlying physical mechanism needed to solve the task, leading to a significant drop in performance. Furthermore, improvements from model scaling quickly saturate, strong general reasoning does not reliably translate to creative affordance discovery, and common inference-time strategies such as Chain-of-Thought yield limited gains. These results suggest that creative tool use remains a major challenge for current models, and that CreativityBench provides a useful testbed for studying this missing dimension of intelligence, with potential implications for planning and reasoning modules in future agents.
An AI might pick a brick. But figuring out to use its corner, not its flat face, to chip away at something? Explaining the physics of that concentrated force?
That’s where it fails. Simply making models bigger hits a wall. Strong performance on standard reasoning tests means nothing here.
Even prompting for step-by-step logic yields minimal gains. This is a fundamental failure in seeing physical potential. CreativityBench doesn't solve it.
The benchmark simply measures the gap with brutal accuracy. Before we can build agents that improvise, we need to see how deeply they don't understand the stuff right in front of them.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Self-Attentive Meta-Optimizer Adds Gradient Alignment and Group-Adaptive Rates
From Archives Local edits in LLM-driven NAS can trigger broader performance shifts
Common Questions Answered
What is the 4K-entity affordance KB that CreativityBench uses?
The 4K-entity affordance knowledge base contains 4,000 objects with over 150,000 annotations describing what different parts of those objects can actually do. This comprehensive dataset enables the benchmark to test whether LLMs can identify alternative uses for objects beyond their intended purpose.
How does CreativityBench differ from standard LLM reasoning benchmarks?
Unlike standard reasoning tests that measure logical thinking, CreativityBench specifically tests creative problem-solving by requiring models to invent novel uses for objects rather than identify their intended applications. The benchmark reveals that strong performance on traditional reasoning tests does not translate to success in physical creativity tasks.
Why do current LLMs fail at tasks like using a brick's corner instead of its flat face?
Current LLMs struggle with understanding the physics of concentrated force and how to apply objects in unconventional ways, such as using a brick's corner to chip away at something. This represents a fundamental failure in perceiving physical potential and spatial affordances that goes beyond what scaling model size or step-by-step prompting can address.
What do the CreativityBench results reveal about scaling larger AI models?
The benchmark demonstrates that simply making models bigger hits a wall when it comes to creative physical reasoning tasks. Even advanced prompting techniques like requesting step-by-step logic yield only minimal gains, indicating that creativity requires something beyond increased model scale.
Further Reading
- Evaluating Agent Creative Reasoning via Affordance-Based Tool Use — ArXiv
- Towards Advancing Creative Intelligence of Language Model Agents — ACL Anthology
- CreativeBench: Benchmarking and Enhancing Machine Creativity in Code Generation — ArXiv
- Creation-MMBench: Assessing Context-Aware Creativity in Multimodal Models — GitHub/Open-Compass
