36 Strong Simple Baselines Assembled From Nature,...

A team of researchers has built a system that uses large language models to generate baseline comparisons for deep learning papers, pulling test cases from 36 studies published in Nature, Science, and journals in the Cell family. The motivation is a problem that has dogged machine learning's spread into biology, chemistry, and physics for years: a new method looks impressive until someone checks whether a far simpler approach gets the same result. That check often doesn't happen before publication.

Peer review, the researchers note, hasn't kept pace with the volume of papers now claiming deep learning breakthroughs in the physical and life sciences. Reviewers with limited time can miss that a proposed neural network solves a problem no better than linear regression or a basic statistical model. The result is a research record padded with claims that don't hold up once someone builds the missing comparison.

Automating that comparison, rather than relying on reviewers to catch it after the fact, is the core idea tested here. Whether a language model can reliably construct a baseline strong enough to actually test a paper's claims, across 36 real published studies, is the question the work sets out to answer.