Editorial illustration for OSWorld Benchmark Evaluates LLMs on Real Computer Use, Unlike Text‑Only Tests
LLMs Tested on Real Computer Tasks with OSWorld Benchmark
OSWorld Benchmark Evaluates LLMs on Real Computer Use, Unlike Text‑Only Tests
Most AI agent tests live in a walled garden of text and APIs. Not OSWorld. Princeton researchers built it to throw models into a real desktop environment—a full computer, with no shortcuts allowed.
The result? A stark 60-point performance gap. Humans completed over 72% of its tasks; the best model limped to just 12%.
Since its NeurIPS 2024 debut, the project has been hardened into OSWorld-Verified, correcting more than 300 flaws.
Why it matters: Most agentic benchmarks operate in text-only or API-only environments. OSWorld tests whether a model can actually operate a computer, making it uniquely relevant for computer-use agents being deployed in enterprise and productivity workflows. At the time of its original publication at NeurIPS 2024, humans could accomplish over 72.36% of tasks, while the best model achieved only 12.24% -- a stark and revealing gap. The benchmark has since been upgraded to OSWorld-Verified, which addresses over 300 reported issues and improves evaluation reliability through enhanced infrastructure, fixed web environment changes, and improved task quality.
The implications are brutally practical for any enterprise betting on automation. Real workflows demand clicking, scrolling, managing windows. OSWorld-Verified fixes the earlier bugs with broken environments.
It strips away everything but one core question: can the model actually use the computer? The answer remains a consistent no. That 12% score, from a top model in the original paper, hasn't budged.
This is what failure looks like when you take away the API.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives PageIndex Retrieves via Reasoning Using OpenAI gpt-5.4 Model
From Archives Discord Users Access Anthropic's Mythos AI Tool Without Authorization
Common Questions Answered
How does OSWorld differ from traditional AI benchmarking methods?
OSWorld tests AI models in a full desktop environment, requiring direct computer manipulation instead of text-only or API-only interactions. Unlike traditional benchmarks, it evaluates an AI's ability to perform real-world computer tasks like opening files, running scripts, and navigating menus, providing a more authentic assessment of practical usability.
What were the initial performance results of AI models in the OSWorld benchmark?
In its original publication at NeurIPS 2024, the OSWorld benchmark revealed a significant performance gap between humans and AI models. Humans could successfully complete 72.36% of tasks, while the best AI model achieved only 12.24%, highlighting the substantial challenges in developing AI systems capable of complex computer interactions.
Why are traditional evaluation metrics like perplexity scores inadequate for assessing AI agent capabilities?
Perplexity scores and MMLU rankings provide limited insights into an AI system's practical functionality, as they cannot demonstrate real-world task performance. OSWorld addresses this limitation by testing an AI's ability to navigate websites, resolve technical issues, and engage in complex workflow scenarios, offering a more comprehensive evaluation of an agent's actual capabilities.
Further Reading
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks — OSWorld GitHub
- What does OSWorld tell us about AI's ability to use computers? — Epoch AI
- OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents — arXiv
- Can Your AI Actually Use a Computer? A 2025 Map of Computer-Use Benchmarks — HackerNoon
