Editorial illustration for Terminal-Bench 2.0 Unveils Harbor: Universal Container Agent Testing Platform
Harbor: Universal Container Agent Testing Breakthrough
Terminal-Bench 2.0 launches with Harbor, testing any container-installable agent
Testing AI agents is a mess. Every team runs a different set of bespoke checks, and no one agrees on what passing means. Terminal-Bench just released version 2.0 to fix that, launching a companion tool called Harbor designed to test any piece of software you can shove into a container.
It’s an attempt to standardize the chaos. The idea is to give developers one flexible framework to evaluate agents across different architectures, instead of a dozen incompatible scripts.
Designed to generalize across agent architectures, Harbor supports: Evaluation of any container-installable agent Scalable supervised fine-tuning (SFT) and reinforcement learning (RL) pipelines Custom benchmark creation and deployment Full integration with Terminal-Bench 2. Harbor was used internally to run tens of thousands of rollouts during the creation of the new benchmark. It is now publicly available via harborframework.com, with documentation for testing and submitting agents to the public leaderboard.
Early Results: GPT-5 Leads in Task Success Initial results from the Terminal-Bench 2.0 leaderboard show OpenAI's Codex CLI (command line interface), a GPT-5 powered variant, in the lead, with a 49.6% success rate -- the highest among all agents tested so far. Close behind are other GPT-5 variants and Claude Sonnet 4.5-based agents. Top 5 Agent Results (Terminal-Bench 2.0): Codex CLI (GPT-5) -- 49.6% Codex CLI (GPT-5-Codex) -- 44.3% OpenHands (GPT-5) -- 43.8% Terminus 2 (GPT-5-Codex) -- 43.4% Terminus 2 (Claude Sonnet 4.5) -- 42.8% The close clustering among top models indicates active competition across platforms, with no single agent solving more than half the tasks.
Submission and Use To test or submit an agent, users install Harbor and run the benchmark using simple CLI commands. Submissions to the leaderboard require five benchmark runs, and results can be emailed to the developers along with job directories for validation. harbor run -d terminal-bench@2.0 -m "<model>" -a "<agent>" --n-attempts 5 --jobs-dir <path/to/output> Terminal-Bench 2.0 is already being integrated into research workflows focused on agentic reasoning, code generation, and tool use.
According to co-creator Mike Merrill, a postdoctoral researcher at Stanford, a detailed preprint is in progress covering the verification process and design methodology behind the benchmark. Aiming for Standardization The combined release of Terminal-Bench 2.0 and Harbor marks a step toward more consistent and scalable agent evaluation infrastructure.
The early leaderboard results tell a familiar story. GPT-5 is on top. But not by much.
No model solves more than half the tasks, and the top five are clustered within seven percentage points. It’s a tight race where nobody is winning decisively.
Harbor’s real value might be in its mundane internal track record. The team used it to run tens of thousands of rollouts while building the new benchmark. That’s a good sign. It suggests the tool is built for actual work, not just a press release.
Now it’s public. Anyone can install it and start testing. The promise is a simpler, unified process for a job that’s usually fragmented and annoyingly complex. Whether it becomes a standard depends entirely on if developers find it useful, or just another piece of infrastructure to manage.
Related Reading
Business & Startups Nordic pilot adds Gemini for Education, NotebookLM to boost AI literacy
Business & Startups Kling launches Video O1, all-in-one model with MVL bridge using transformer
Business & Startups DeepSeek Seeks More Capital Weeks After USD 7B Funding Round
From Archives OpenAI CEO Sam Altman projects USD 20 billion revenue, backs US chip push
From Archives BBVA scales AI to 11,000 staff, saving 3 hrs weekly per employee
Common Questions Answered
How does Harbor support testing container-based AI agents across different architectures?
Harbor provides a universal testing platform that can evaluate any container-installable agent, regardless of its underlying architectural framework. The tool supports scalable supervised fine-tuning (SFT) and reinforcement learning (RL) pipelines, enabling comprehensive and flexible agent testing across diverse environments.
What key capabilities does Harbor offer for researchers and developers?
Harbor enables custom benchmark creation and deployment, allowing developers to design specialized testing environments for container agents. The platform was internally used to run tens of thousands of rollouts during benchmark development, demonstrating its robust testing capabilities and versatility.
Where can developers access the Harbor testing framework?
The Harbor framework is publicly available via harborframework.com, which provides comprehensive documentation for testing and submitting agents to the public platform. Developers can explore the framework's features and integration methods through the official website.
Further Reading
- Introducing Terminal-Bench 2.0 and Harbor — Terminal-Bench News
- Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits — YouTube (Interview with Terminal-Bench creators)
- Terminal-Bench - Vals AI — Vals AI
