Editorial illustration for SciConBench launches with 9.11K questions to test AI scientific synthesis
SciConBench launches with 9.11K questions to test AI...
Forget whether AI can write your emails. The real question is whether it can do science. A new, brutally difficult benchmark called SciConBench makes it clear that the answer, for now, is a hard no.
It presents models with 9,110 questions pulled directly from published systematic reviews. Each one comes with a verified, expert-written conclusion. The test is not to find a pre-digested answer, but to force the AI to build that conclusion itself from raw materials.
The scoring is merciless. It breaks each expert conclusion into atomic facts, then checks the AI’s output for correctness and completeness using precision and recall. To stop models from cheating with memorized data, a clean-room system called SciConHarness forces them to interact with the web in a controlled, leak-proof environment.
We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement.
Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available.
Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.
The results are dismal. In that clean room, the best-performing frontier agent managed a factual F1 score of 0.337. That’s not a grade.
It’s an indictment. The performance plunge compared to unconstrained testing proves a quiet scandal: data leakage has been flattering AI’s abilities for years.
Worse, tools people might actually use, like Google AI Overview or OpenEvidence, fail in public. They churn out incomplete or logically contradictory summaries even when the correct, complete answer is sitting right there in the text they retrieved. This isn't about needing more data.
It’s a structural failure. Real synthesis requires breaking a problem apart, verifying each piece, and assembling them into a new, coherent whole without inventing comforting lies. Current models cannot do that.
They interpolate. They confabulate. SciConBench is the first honest report card, and every model failed.
Building something that can actually reason, rather than just recite, will require a different kind of machine altogether.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Language Agents Self‑Gate Clarification: Mandatory vs Opportunistic Modes
From Archives Study Defines Privacy-Utility Frontier for Agent Memory via PR and AER
Common Questions Answered
What is SciConBench and how does it test AI scientific synthesis?
SciConBench is a benchmark containing 9,110 questions extracted directly from published systematic reviews, each paired with expert-verified conclusions. Rather than testing whether AI can find pre-digested answers, it forces AI models to build conclusions themselves from raw materials, creating a rigorous test of genuine scientific reasoning capabilities.
What were the results of SciConBench testing on frontier AI models?
The results were dismal, with the best-performing frontier agent achieving a factual F1 score of only 0.337. This poor performance reveals that data leakage has been artificially inflating AI abilities in previous unconstrained testing scenarios.
How do practical AI tools like Google AI Overview perform on SciConBench?
Tools people actually use in practice, such as Google AI Overview and OpenEvidence, fail publicly on SciConBench by producing incomplete or logically contradictory summaries. This demonstrates that current AI systems struggle to synthesize scientific information accurately even when the correct and complete information is available.
What does the data leakage problem reveal about previous AI capability assessments?
The significant performance drop when AI models are tested in SciConBench's controlled environment compared to unconstrained testing proves that data leakage has been systematically flattering AI's abilities for years. This indicates that previous benchmark results may not accurately reflect true AI capabilities in real-world scientific synthesis tasks.
