AI Giants Promise Publishers New Way to Block Some Scraping

OpenAI, Google, Microsoft, and Anthropic have pledged to support a single standardized signal that lets publishers opt out of AI training scrapes, but the tool won't fully work until 2027 and content remains scrapable by default until publishers change their settings.

News source: Google /  https://www.channelnewsasia.com/international

The biggest names in artificial intelligence have promised publishers a new way to block their content from being scraped to train AI models. The fine print matters more than the headline.

OpenAI, Google, Microsoft, and Anthropic announced Friday they will adopt a shared "policy token" that publishers can place in their robots.txt file, signaling a preference to opt out of AI training across multiple companies at once. Currently, publishers must configure separate blocking commands for every individual AI crawler.

The announcement comes as publishers face a surge in bots scraping their articles to feed AI models. The New York Times sued OpenAI and Microsoft in December 2023, accusing them of using millions of articles without authorization or payment. Cloudflare has blocked AI bots by default on new customer sites since July 2025.

The Catch: Default Is Still Open

The new setting will apply only to future training of models. It does not prevent bots from crawling websites otherwise, and content scraping remains allowed by default until a publisher actively changes the setting, Microsoft said in a blog post.

Microsoft will implement the signal beginning February 1, 2027. Google plans to adopt it sometime in 2027. OpenAI said it will roll it out "in the coming months," while Anthropic said it has "already started."  Meta's communications head Andy Stone expressed support for the effort but did not confirm whether the company would honor it.

Here is the question The Silicon Review raises. The AI companies that built their models by scraping the open web are now offering publishers a button to stop the scraping. But the button does not work for months, the default is still "scrape everything," and the companies offering it are the same ones that profited from the content they took. Is this a genuine concession or a public relations move designed to slow down the lawsuits?

FAQ:

Q: What is the new AI scraping opt-out tool?
A: It is a shared policy token placed in a website's robots.txt file that signals a publisher's preference to opt out of AI training across multiple companies at once.

Q: When will the new AI scraping tool take effect?
A: Microsoft begins February 1, 2027, Google sometime in 2027, OpenAI in the coming months, and Anthropic has already started.

Q: Does the opt-out block all AI crawling?
A: No. It applies only to future AI training. Bots can still crawl the site, and content remains scrapable by default until a publisher changes the setting.

Q: Why did the AI companies agree to this?
A: The announcement follows growing criticism and legal battles from news publishers, including the New York Times lawsuit filed in December 2023.

Q: Will opting out affects a publisher's visibility in AI search results?
A: Yes. Microsoft and Google said opting out of AI training could reduce the chances a publisher's content surfaces in AI apps and experiences.