9 LLMs Tested on 2,005 Oncology Decisions

Researchers ran nine frontier language models through 2,005 oncology decision points and found a pattern that no single model's benchmark score would have predicted. The test wasn't trivia. It asked models to follow guideline pathways, judge when to escalate care, and commit to a next step under the kind of uncertainty that defines actual clinical practice, not the multiple-choice format of medical licensing exams.

That distinction matters because most existing LLM benchmarks measure factual recall, a skill where frontier models already perform well. What those benchmarks don't capture is whether a model will actually commit to the correct action once it has identified it, or whether it hedges, stalls, or defers at the exact moment a clinician needs a clear recommendation. The study set out to check whether these decision-path failures are model-specific quirks or something shared across architectures, and whether stacking multiple models together could paper over the gaps.

The answer bears directly on how health systems are currently thinking about deploying LLMs in oncology settings, where the assumption has been that better models, or more models working in concert, would eventually close the safety gap.