Editorial illustration for Yahoo and Ziff Davis Back New AI Licensing Standard for Web Content Scraping
Web Publishers Unite to Set AI Content Licensing Rules
RSL 1.0 AI Licensing Standard launches with Yahoo, Ziff Davis support
Everyone in AI says they love publishers. They love them like a bear loves a picnic basket. The actual business model has been to vacuum up everything ever written, ignore decades-old rules, and hope nobody lawyers up. That party might be ending.
RSL 1.0 is a new technical standard. It is basically a toll booth for AI scrapers. Backed by Yahoo, Ziff Davis, and O’Reilly Media, it gives publishers a formal way to demand payment for the content AI firms have been taking.
The crucial part is the enforcers. Cloudflare, Akamai, and Fastly have signed on. The standard itself is just text in a file.
These companies are the ones who can actually cut off the tap at the network level.
RSL 1.0 helps publishers outline how AI companies should pay for the content they scrape across the web. The RSL Collective announced the standard in September with backing from Yahoo, Ziff Davis, and O'Reilly Media. It's an expansion of the robots.txt file, which outlines the parts of a website a web crawler can access.
Though RSL alone can't block AI scrapers that don't pay for a license, the web infrastructure providers that support the standard can -- a list that now includes Cloudflare and Akamai, in addition to Fastly. RSL's 1.0 release lets publishers block their content from AI-powered search features, like Google's AI Mode, while maintaining a presence in traditional search results.
The immediate practical effect is publishers can now block their content from being used in AI search summaries, like Google's new AI Mode, without disappearing from normal search results. It is a technical distinction with major financial stakes. You can keep your traffic from being cannibalized by a chatbot answer while still being found by humans.
Will every AI company suddenly start paying? No. Some will ignore it.
Others will look for unprotected sites. But for the first time, the biggest gatekeepers of web traffic are siding with publishers, not the scrapers. This is not a legal victory.
It is an infrastructural one. The free lunch now comes with a suggested price tag, posted by the people who control the doors.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives AgentLLM: Open-source browser tool runs local LLM agents for tasks
From Archives ChatGPT adds Adobe Express to create and edit designs, photos, PDFs
Common Questions Answered
What is the RSL 1.0 standard and how does it aim to help content creators?
RSL 1.0 is a new web content licensing standard designed to give publishers more control over how AI platforms use their digital content. The standard provides a framework for AI companies to pay for web content they scrape, extending the traditional robots.txt protocol to create clearer guidelines for digital content licensing.
Which major companies are supporting the RSL 1.0 initiative?
Yahoo, Ziff Davis, and O'Reilly Media are backing the RSL 1.0 standard, demonstrating significant industry support for the new content licensing framework. Web infrastructure providers like Cloudflare have also joined the initiative, which could help enforce the proposed licensing guidelines.
How does RSL 1.0 differ from the traditional robots.txt protocol?
While robots.txt traditionally outlined which parts of a website web crawlers could access, RSL 1.0 goes further by establishing a licensing mechanism for AI content scraping. The new standard provides a mechanism for publishers to potentially receive compensation for their content used by AI platforms, creating a more structured approach to digital content access.
Further Reading
- New RSL Web Standard and Collective Rights Organization Automate Content Licensing for the AI-First Internet and enable Fair Compensation for Millions of Publishers and Creators — RSL Collective
- RSL: A New AI Licensing Standard — Shelly Palmer
- New RSL Standard Aims to Stop Unpaid AI Content Scraping — Tech.co
- Really Simple Licensing (RSL) for AI — UBC Wiki
