AI Giants Back Single Web Crawler Opt-Out Standard

OpenAI, Google, Microsoft, and Anthropic have agreed to back a single standard that lets publishers opt out of AI scraping in one move, rather than filing separate requests with each company. The change addresses a problem that's been building for years: every time a new AI crawler shows up, site owners have had to dig into their robots.txt files and add instructions for that specific bot. Miss one, and your content gets pulled into a training set anyway.

Kevin Kelley, a technical lead for standards at Anthropic, proposed the fix through the Internet Engineering Task Force. It's a shared signal, something like a "policy token," that sits inside the robots.txt file sites already use to tell crawlers what they can and can't touch. Instead of writing out rules for OpenAI's bot, then Google's, then Microsoft's, then Anthropic's, a publisher sets one preference that all four companies say they'll honor.

The timing isn't random. Publishers have spent the last couple of years trying to slow the flood of AI crawlers hitting their servers, with mixed results, from licensing schemes like RSL to Cloudflare's default blocking. This is the biggest coordinated response yet from the AI companies themselves.