Editorial illustration for Harness-1 20B Model Beats GPT-5.4, Curates Top 8 Fairness‑Rated Results
Harness-1 20B Model Beats GPT-5.4, Curates Top 8...
Harness-1, a 20-billion-parameter AI subagent built for retrieval, now outperforms OpenAI’s GPT-5.4 model. The improvement came from a policy tweak. Instead of deleting weak results, its system saves and reranks the top eight results tagged with a fairness score, refining the dataset upward.
A live GPT-5.4 model generated training data from diverse queries, but only 899 clean episodes survived filtering. Those episodes taught the smaller agent how to call tools and structure actions. Without a specific diversity bonus, the agent stalled, recycling similar searches and achieving a recall score of 0.53.
Adding the bonus pushed recall to 0.60, as the agent learned to use additional tools like grep_corpus and verify.
Have I seen this chunk before? RL ends up optimizing both, and honestly, they don’t share the same learning dynamics.
The method shows refining an existing dataset can be more effective than building one from nothing. Training on those 899 episodes from SEC document queries produced policies that worked across eight different benchmark domains. The key was the diversity bonus.
It broke a loop of repetitive searches and prompted the agent to use a broader set of tools. That single change lifted its curated recall score from 0.53 to 0.60, letting the 20B-parameter Harness-1 beat its far larger teacher model, GPT-5.4, at search.
Related Reading
Browse all OpenAI news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source
Related Topic OpenAI Calls Apple's Lawsuit 'Rotten to Its Core' in Rebuttal
Policy & Regulation
From Archives DFlash drafts whole token blocks, achieving 15× throughput on NVIDIA Blackwell
From Archives RIFT-Bench Introduces Graph-Driven Dynamic Red-Teaming for Agentic AI
From Archives OpenAI's GPT-5.5-Cyber Beats Anthropic Mythos, Starts Patching Initiative
From Archives Alibaba AI video model climbs to #2 as Sora withdrawal warns firms
Business & Startups
Common Questions Answered
How does Harness-1's fairness-rating system improve performance over GPT-5.4?
Instead of deleting weak results, Harness-1 saves and reranks the top eight results tagged with a fairness score, which refines the dataset upward. This approach allowed Harness-1's 20-billion-parameter model to outperform OpenAI's much larger GPT-5.4 model by lifting its curated recall score from 0.53 to 0.60.
What role does the diversity bonus play in Harness-1's retrieval performance?
The diversity bonus breaks the loop of repetitive searches and prompts the agent to use a broader set of tools across different queries. This single change was key to improving Harness-1's performance, as it enabled the model to train effectively on 899 episodes from SEC document queries and generalize across eight different benchmark domains.
Why is refining an existing dataset more effective than building one from scratch for Harness-1?
The article demonstrates that refining an existing dataset through policy tweaks and fairness-based reranking produces better results than creating a new dataset from nothing. By training on curated, diverse query episodes with fairness scores, Harness-1 achieved superior performance metrics compared to larger models, proving that dataset quality and diversity matter more than starting from zero.
What specific improvement did the policy tweak achieve in Harness-1's curated recall score?
The policy tweak of saving and reranking the top eight fairness-rated results instead of deleting weak results increased Harness-1's curated recall score from 0.53 to 0.60. This 0.07-point improvement was significant enough to allow the 20-billion-parameter Harness-1 to outperform GPT-5.4, demonstrating the effectiveness of the fairness-rating approach.
Further Reading
- Researchers trained an open source AI search agent, Harness-1, that outperforms GPT-5.4 on recalling relevant information — VentureBeat
- Harness-1 Open Source AI Agent Outperforms GPT-5.4 - Meteora Web — Meteora Web
- Which One Actually Moves Your Agent's Benchmark Score? — MindStudio
- Are the New GPT-OSS Models Any Good? We put them to the test. — DataRobot
- In harmony with gpt-oss — arXiv
