OpenAI, Google, and Microsoft will honor a new ‘policy token’ signal in robots.txt files to respect publisher AI training preferences. The shared signal, proposed by Anthropic’s Kevin Kelley, allows publishers to set AI training rules for multiple providers at once. Publishers currently must maintain separate rules for each AI company, a process the new standard aims to simplify.
OpenAI, Google, Microsoft, and Anthropic have announced they will restrict AI crawlers from accessing sites with the shared token. The announcement comes as publishers struggle with unauthorized AI crawlers scraping content for model training. The standard is part of ongoing work by the Internet Engineering Task Force (IETF) AI Preferences working group. Meta communications head Andy Stone expressed support for the common signal but did not confirm if the company will honor it.
