Editorial illustration for OpenDiscoveryTrace Records 124 AI Scientist Workflows With 9-Step Detail
OpenDiscoveryTrace Maps 558 AI Science Workflows
OpenDiscoveryTrace Records 124 AI Scientist Workflows With 9-Step Detail
A new dataset out this week puts numbers on something researchers evaluating AI scientists have mostly had to guess at: how the models actually got to their answers. OpenDiscoveryTrace contains 558 complete trajectories from autonomous AI agents running scientific workflows, each one broken into a 9-step process trace covering hypothesis generation, tool calls, and error recovery, not just the final code or paper produced at the end.
The gap this fills is specific. Most benchmarks for autonomous science agents score outputs, generated hypotheses, experiment code, written results, and stop there. If a model stumbles into a correct answer through a broken chain of reasoning, current evaluation methods can't tell the difference between that and a genuinely sound methodology. That matters for anyone trying to audit whether an AI system can be trusted to run unsupervised experiments, and it matters for governance frameworks that need more than pass/fail output checks.
OpenDiscoveryTrace's authors built the dataset around 124 recorded workflows spanning multiple models and task types, then defined five benchmark tasks with baselines ranging from logistic regression to Transformer models, released publicly under a CC BY 4.0 license.
Each trajectory records a structured 9-field-per-step trace---including thoughts, tool calls, observations, errors, revision triggers, and self-reported confidence---as models execute 124 scientific tasks spanning drug discovery, materials science, genomics, and scientific literature analysis.
Why this matters
For anyone building or evaluating autonomous research agents, OpenDiscoveryTrace is a useful corrective. Most benchmarks in this space grade the paper or the code at the end and ignore how the system got there, which means a lucky guess and a sound chain of reasoning score identically. Recording nine fields per step, thoughts, tool calls, observations, errors, revision triggers, confidence, across 558 trajectories and 124 tasks in drug discovery, materials science, genomics, and literature analysis, gives researchers something they've largely lacked: a way to actually audit method, not just outcome.
That matters for founders selling "AI scientist" products, because it becomes possible to check whether a system revises after an error or just stumbles forward. It matters for researchers building evaluation frameworks, because process-level data lets you separate systematic reasoning from noise. We'd want to see how the trace format holds up against adversarial or messier real-world tasks before treating it as a standard, but as a dataset for diagnosing failure modes rather than just scoring final answers, it fills a real gap.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Robotaxi Leaders Build With NVIDIA’s Physical AI Tools
From Archives Deepmind Eased Ban on Staff Discussing AI Risks After Internal Pushback
Common Questions Answered
What are the 9 steps recorded in each trajectory within the OpenDiscoveryTrace dataset?
Each trajectory records a structured 9-field-per-step trace that includes thoughts, tool calls, observations, errors, revision triggers, and self-reported confidence as models execute scientific tasks. This detailed process tracing captures the complete reasoning path of autonomous AI agents rather than just their final outputs, providing comprehensive insight into how AI scientists arrive at their conclusions.
How many scientific workflows and tasks does OpenDiscoveryTrace contain?
OpenDiscoveryTrace contains 558 complete trajectories from autonomous AI agents running 124 scientific tasks. These tasks span multiple domains including drug discovery, materials science, genomics, and scientific literature analysis, providing a diverse dataset for evaluating AI scientist workflows.
Why is process-level tracing important for evaluating autonomous research agents?
Most existing benchmarks only evaluate the final paper or code produced by autonomous research agents, meaning a lucky guess and sound reasoning score identically. OpenDiscoveryTrace addresses this gap by recording the complete 9-step process, allowing researchers to distinguish between agents that arrive at correct answers through proper methodology versus those that succeed through chance.
What specific information does each step in an OpenDiscoveryTrace trajectory capture?
Each step captures seven key data points: the model's thoughts, tool calls made, observations received, errors encountered, revision triggers that prompted changes, self-reported confidence levels, and other structured fields. This comprehensive per-step documentation enables detailed analysis of how autonomous AI agents make decisions and recover from mistakes during scientific workflows.
Further Reading
- Process Traces for Evaluating AI Scientist Workflows - ICML 2026
- Evaluating AI Scientist Workflows with OpenDiscoveryTrace - Gworky
- Graph of Trace: Visualizing Execution Traces of Scientific Agents - arXiv
- TRACES puts AI to the test for the next frontier of AI Discovery - itnews.asia
- Apodex Launches TRACES, a Benchmark for Scientific Discovery - PR Newswire
