Editorial illustration for Yahoo and Ziff Davis Back New AI Licensing Standard for Web Content Scraping
Web Publishers Unite to Set AI Content Licensing Rules
RSL 1.0 AI Licensing Standard launches with Yahoo, Ziff Davis support
The web's content gold rush just got a rulebook. As AI companies continue harvesting online information without clear compensation guidelines, publishers are fighting back with a new standard that could reshape digital content licensing.
Yahoo and Ziff Davis are backing an ambitious initiative that promises to give content creators more control over how their material gets used by artificial intelligence platforms. The proposed standard, known as RSL 1.0, represents a potential turning point in the increasingly complex relationship between publishers and AI companies.
Web content has long been a free-for-all, with AI systems scraping websites indiscriminately. But this new framework suggests a more structured approach is emerging - one that could fundamentally change how digital information gets accessed, used, and potentially monetized.
The stakes are high. With AI models consuming massive amounts of online data to train their systems, publishers are seeking a way to protect their intellectual property and ensure fair compensation. The RSL 1.0 standard might just be their most promising solution yet.
RSL 1.0 helps publishers outline how AI companies should pay for the content they scrape across the web. The RSL Collective announced the standard in September with backing from Yahoo, Ziff Davis, and O'Reilly Media. It's an expansion of the robots.txt file, which outlines the parts of a website a web crawler can access.
Though RSL alone can't block AI scrapers that don't pay for a license, the web infrastructure providers that support the standard can -- a list that now includes Cloudflare and Akamai, in addition to Fastly. RSL's 1.0 release lets publishers block their content from AI-powered search features, like Google's AI Mode, while maintaining a presence in traditional search results.
The RSL 1.0 standard represents a significant step for digital publishers seeking compensation in the AI content landscape. Yahoo and Ziff Davis have thrown their weight behind a potential solution that extends the traditional robots.txt protocol, signaling publishers' growing determination to control web content access.
Web infrastructure providers like Cloudflare's support suggests the standard might gain meaningful traction. Still, the licensing framework's effectiveness remains uncertain without universal adoption by AI companies.
The standard's core idea lies in providing a clear mechanism for publishers to define how their content can be used and monetized by AI systems. By expanding on existing web crawling protocols, RSL 1.0 offers a structured approach to what has been an increasingly complex digital rights challenge.
While the standard can't unilaterally block unauthorized scraping, its backing by major media companies indicates a serious industry effort to address content compensation. Publishers now have a framework to potentially negotiate fair terms for their intellectual property in the rapidly evolving AI ecosystem.
Further Reading
- LAION Round 2: Machine-Readable but Still Not Actionable – The Lack of Progress on TDM Opt-Outs (Part 2) - Wolters Kluwer Legal Blog
- What's on tap for news SEO in 2026? - SEO for Journalism
- Poynter and Hacks/Hackers partner to keep fast AI adoption aligned with journalism ethics - Editor & Publisher
- AI OmniBrief – Week 52, 2025 - AI OmniBrief Substack
Related Reading
LLMs & Generative AI Ant Group unveils Ring-1T, first open-source trillion-parameter reasoning model
LLMs & Generative AI ChatGPT Health Event Shows AI Modernizing Dev Workflows, GitLab Unveils Plans
LLMs & Generative AI Gen AI app sessions up fivefold, downloads jump 778% as ChatGPT leads traffic
From Archives AgentLLM: Open-source browser tool runs local LLM agents for tasks
From Archives ChatGPT adds Adobe Express to create and edit designs, photos, PDFs
Common Questions Answered
What is the RSL 1.0 standard and how does it aim to help content creators?
RSL 1.0 is a new web content licensing standard designed to give publishers more control over how AI platforms use their digital content. The standard provides a framework for AI companies to pay for web content they scrape, extending the traditional robots.txt protocol to create clearer guidelines for digital content licensing.
Which major companies are supporting the RSL 1.0 initiative?
Yahoo, Ziff Davis, and O'Reilly Media are backing the RSL 1.0 standard, demonstrating significant industry support for the new content licensing framework. Web infrastructure providers like Cloudflare have also joined the initiative, which could help enforce the proposed licensing guidelines.
How does RSL 1.0 differ from the traditional robots.txt protocol?
While robots.txt traditionally outlined which parts of a website web crawlers could access, RSL 1.0 goes further by establishing a licensing mechanism for AI content scraping. The new standard provides a mechanism for publishers to potentially receive compensation for their content used by AI platforms, creating a more structured approach to digital content access.