Editorial illustration for Harvard study finds OpenAI's o1 and 4o outdiagnose ER doctors in 76‑patient test
Harvard study finds OpenAI's o1 and 4o outdiagnose ER...
Seventy-six patients walked into a Boston emergency room. Two attending physicians examined them, made their calls. So did two AI models from OpenAI, o1 and 4o.
The diagnoses were then handed to a separate pair of doctors, blinded, unware which came from human or machine. The result? At every diagnostic touchpoint, o1 matched or edged out the humans.
The gap widened most where it matters: initial triage, when information is scarce and the stakes are highest.
In one experiment, researchers focused on 76 patients who came into the Beth Israel emergency room, comparing the diagnoses offered by two attending physicians to those generated by OpenAI’s o1 and 4o models. These diagnoses were assessed by two other attending physicians, who did not know which ones came from humans and which came from AI. “At each diagnostic touchpoint, o1 either performed nominally better than or on par with the two attending physicians and 4o,” the study said, adding that the differences “were especially pronounced at the first diagnostic touchpoint (initial ER triage), where there is the least information available about the patient and the most urgency to make the correct decision.”
The numbers are small. The stakes, however, are immense. In a 76-patient test, OpenAI’s models didn’t just keep pace with human doctors, they outperformed them, most decisively at the moment of greatest uncertainty: triage.
That is the point where a split-second decision can cascade into life or death. The AI had less information, yet it made the sharper call. This is not a story about machines replacing physicians.
It is a story about the limits of human cognition under pressure. The ER is a fog of war. Fatigue, bias, and the sheer noise of a chaotic shift degrade even the best-trained mind.
The models, by contrast, are immune to that noise. They do not get tired. They do not get distracted by the next patient’s chart or the overhead page.
They simply compute. The study’s design is elegant in its cruelty: the human judges did not know which diagnosis came from a doctor and which from a machine. That anonymity is the point.
It strips away the prestige of the white coat and forces a raw comparison of output. And the output speaks clearly. At triage, where the stakes are highest and the data thinnest, the AI was sharper.
This does not mean the end of the emergency physician. It means the end of the physician who works alone. The best future for the ER is not human versus machine, but human augmented by machine, a partnership where the AI handles the cognitive load of pattern recognition while the doctor manages the context, the touch, the impossible human variables that no model can yet weigh.
The study is a warning, but also an invitation. The doctors who embrace this tool will not be replaced. They will be elevated.
The ones who ignore it will be left behind.
Related Reading
Browse all OpenAI news →
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Related Topic OpenAI researcher quits, citing distrust over ad‑driven engagement metrics
LLMs & Generative AI
Related Topic OpenAI launches GPT-Image 1.5 with precise editing for enterprise visuals
LLMs & Generative AI
From Archives 2021 EDEN-unbiased quantizer beats 2026 successor in average accuracy
From Archives US benchmark shows China lagging; Deepseek model underperforms private tests
From Archives OpenAI releases Symphony, an open-source spec to let agents self‑manage tasks
Open Source
From Archives Elon Musk and Sam Altman head to court in legal fight over OpenAI’s future
Business & Startups
Common Questions Answered
How did OpenAI's o1 and 4o models perform compared to ER doctors in the Harvard study?
In a 76-patient test conducted at a Boston emergency room, OpenAI's o1 and 4o models matched or outperformed human attending physicians at every diagnostic touchpoint. The AI models were particularly superior during triage, the critical moment where split-second decisions can have life-or-death consequences, despite having access to less information than the human doctors.
What was the study methodology for comparing AI diagnoses to human physician diagnoses?
Two attending physicians examined 76 patients in a Boston emergency room and made their diagnoses, while OpenAI's o1 and 4o models independently provided their own diagnoses. A separate pair of doctors then reviewed all diagnoses while blinded to whether they came from humans or AI, ensuring an unbiased evaluation of diagnostic accuracy.
At what diagnostic stage did the AI models show the most significant advantage over human doctors?
OpenAI's AI models demonstrated their most decisive advantage during triage, the initial assessment phase where patients are prioritized based on the urgency of their condition. This is considered the point of greatest uncertainty and where split-second decisions can have cascading effects on patient outcomes.
What does the Harvard study suggest about the limitations of human doctors in emergency settings?
The study indicates that the performance gap between AI and human doctors reflects the limits of human cognition under pressure rather than a case for machine replacement of physicians. Despite having less information available, the AI models made sharper diagnostic calls, suggesting that cognitive limitations and time pressure affect physician decision-making in emergency situations.
Further Reading
- AI Outperforms Doctors in Emergency Room Tasks, New Harvard Study Shows — Harvard Magazine
- Harvard AI Outperforms Doctors in ER Triage Study — Lets Data Science
- AI outperforms doctors at diagnosing complex emergency room cases — Inshorts
