Editorial illustration for Open AI models keep shrinking cyber capability gap with closed systems
Open AI Models Close Cyber Capability Gap
Vals AI put a handful of leading language models in front of a simulated corporate network, gave each one a public web address and nothing else, and watched what happened. No human help, no hints, just an isolated attack machine and a benchmark called CyberOffense built to mimic a real target. The results narrow a gap that security researchers have been watching for two years: the distance between what closed, tightly controlled AI systems can do to a network and what freely downloadable open-weight models can do.
OpenAI's GPT-6 Sol, built under the company's restricted Daybreak cyber program, topped the field. But Deepseek V4.1 Flash 1017, an open-weight model anyone can download and run, finished close behind at a fraction of the cost. That detail matters because open weights can't be revoked, rate-limited, or monitored the way access to a closed model can. Once they're out, any provider's control over how they get used effectively ends.
Vals ran a second, harder test too, called Lighthouse, to see what happens when an agent is given more time and a bigger objective: breaking into one company and using what it finds there to reach a second.
GPT-6 Sol, part of OpenAI's restricted Daybreak cyber program, posted the best result at 80.8 percent of objectives. Deepseek V4.1 Flash 1017, a freely available open-weight model, isn't far behind at 72.8 percent and costs only about $3 per run instead of roughly $22. Once a model's weights are public, no provider can control or restrict how it's used, Vals researchers say.
Why this matters
For anyone building or deploying agentic systems, the Vals numbers are a warning label, not a curiosity. An 80.8 percent clear rate on GPT-6 Sol against simulated corporate networks shows these agents can already chain exploits with zero human steering, and Deepseek V4's showing proves that capability isn't locked behind restricted programs like Daybreak. Open models closing the gap means the cost of running a credible intrusion attempt drops for anyone with API access, not just well-funded labs.
Vals' own framing, that short benchmarks underestimate real potential, should make founders nervous about how their red-team results get reported internally. The flip side is real too: the same agent architecture that finds a path through a network can be pointed at your own infrastructure first. Security teams that aren't already running these models against their own systems are ceding a head start to whoever runs them first with worse intentions.
Expect procurement conversations, vendor risk assessments, and incident response plans to start referencing benchmarks like CyberOffense directly, because "which model can break in" is becoming a line item, not an abstract research question.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Timestral, t0, Tabby, and Morpheus-2.0 Compared on GIFT-Eval
From Archives Qwen3.8-27B: 22% propaganda; Western models below 1%; fine-tuning strips layer
Common Questions Answered
What did Vals AI discover when testing open and closed language models against the CyberOffense benchmark?
Vals AI found that open-weight models like Deepseek V4.1 Flash are narrowing the cyber capability gap with closed systems, achieving 72.8 percent of objectives compared to GPT-6 Sol's 80.8 percent. This represents a significant closing of the two-year gap that security researchers have been monitoring between what closed, tightly controlled AI systems and freely downloadable open models can accomplish in network attacks.
How much more cost-effective is Deepseek V4.1 Flash compared to GPT-6 Sol for running cyber attacks?
Deepseek V4.1 Flash costs approximately $3 per run compared to roughly $22 for GPT-6 Sol, making it significantly more affordable while still achieving strong performance on the CyberOffense benchmark. This dramatic cost reduction means that actors with basic API access can now conduct credible intrusion attempts without needing expensive restricted programs.
Why is the public availability of open-weight model weights a security concern according to Vals researchers?
Once a model's weights are public, no provider can control or restrict how the model is used, meaning open-weight models like Deepseek cannot be limited to benign applications. This lack of control contrasts with closed systems like GPT-6 Sol, which are part of restricted programs like OpenAI's Daybreak that can enforce usage restrictions.
What capability does the 80.8 percent clear rate on GPT-6 Sol demonstrate about current AI agents?
The 80.8 percent clear rate shows that these AI agents can already chain exploits together with zero human steering against simulated corporate networks. This demonstrates that agentic systems have reached a level of autonomous capability that poses a genuine security threat, serving as a warning label for anyone building or deploying such systems.
Further Reading
- Free Chinese AI Models Are Getting Scary Good at Hacking - WIRED
- CyberOffense Leaderboard and Methodology - Vals AI
- We burned 11.7bn tokens to find the best cyber AI model - Aikido
- CTI-REALM: A new benchmark for end-to-end detection rule generation with AI agents - Microsoft Security
- Vals finds open AI models closing the hacking gap with gated systems - CryptoBriefing
