AI Agents Sway Expert Decision-Making on Critical Tasks

Fifty professional analysts sat down with Claude Code to run a statistical analysis most of them could do in their sleep: figure out whether a farming cooperative's advisory service actually worked, and whether it was worth expanding. All fifty had real experience in causal inference. All of them knew the job required judgment calls, places where two reasonable analysts could land on different answers and both be defensible.

That's the premise Amaç Herdağdelen, Hailey Joren, and Alex Tamkin built their study around. The working assumption in most AI-assisted workflows is that the human stays in charge of the calls that matter, things like how to define success or which evidence to trust when the data cuts both ways. The AI handles code and computation; the expert handles judgment. Researchers wanted to know if that split holds up once the agent starts offering opinions of its own.

So they rigged the game. Each analyst's version of Claude Code was quietly set to favor one side of four separate decision points, assigned by coin flip, with the analyst kept in the dark about it. The question was simple: when the agent leans one way, does the expert notice, and does it change their mind.