Four AI Makers Back One ‘No Training’ Signal

SAN FRANCISCO, Oct. 30 — Google,Google, OpenAI, Microsoft and Anthropic announced support on Friday for a shared signal in a website’s robots.txt file expressing a preference against the use of its pages to train generative AI models. Until now, OpenAI said, a site that wanted to refuse had to identify each company’s crawlers and keep separate rules for every one.

It is a preference, not a license. Microsoft wrote that its new preferences “do not establish licensing or payment arrangements,” and the standards draft behind the signal lets private agreements override what a site has said. For publishers, one shared rule could do the work of several. Nothing in it pays them.

The simplest refusal takes two lines of robots.txt. User-Agent: __usage_ai-training names the purpose, and Disallow: / beneath it refuses training for a whole site; a path in place of the slash refuses it for part of one. It comes from an Internet-Draft that Kevin Kelley, a technical lead for standards at Anthropic, as The Verge described him, filed on Friday with the Internet Engineering Task Force.The Working GroupThe draft is offered to the task force’s AI Preferences working group. Its token takes its meaning from the group’s vocabulary draft, in which AI training means using content to change the learned parameters of a generative AI model.

Microsoft says it will begin honoring the signal on Feb. 1, 2027, as it crawls; Google intends to support it in early 2027; OpenAI, which said it “will honor” it, aims to implement it “in the coming months”; and Anthropic says it has started implementing it. Mr. Kelley wrote to the working group’s mailing list on Friday that “Anthropic has already started implementing support for this draft,” and that it would take “some time to add support across our services.”

Google-Extended is the AI training control Google has offered since 2023. Where a site uses the shared signal to refuse training and does not name Google-Extended, Google plans to interpret that “as if the publisher had disallowed Google-Extended,” wrote John Mueller, a Google search advocate. By Google’s description, Google-Extended also governs grounding, the feeding of content from its search index to a model as it answers, in Gemini Apps and on its Vertex AI service, though not inclusion in Google Search. Read together, Google’s two descriptions mean that a refusal through the shared signal would also keep a site’s pages out of those grounding uses, unless the site names Google-Extended separately.

Training and Answers

Microsoft splits the two. Beside the shared training preference it added a page-level preference of its own for grounding, with a new setting, DisallowExtended, that allows grounding only in Microsoft-operated products such as Copilot and Microsoft 365, not in the grounding services Microsoft provides to third parties. Neither preference changes a site’s eligibility for Bing’s search results. They replace Microsoft’s older NoArchive and NoCache tags, which it will stop reading as a preference for training or grounding after March 1, 2027. Sites using those tags that want to keep content out of grounding must, before then, also record a temporary preference in Bing Webmaster Tools, which covers each page until it is recrawled or until Sept. 30, 2027, whichever comes first. “Neither choice guarantees inclusion, traffic, or payment,” it wrote.

Neither choice guarantees inclusion, traffic, or payment.

The signal is not a rule about crawling: Mr. Kelley called it “a preference about use, not a traffic-management signal.” The new mechanism does not count a blanket User-Agent: * group as a preference, so under it “a robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all,” he wrote. Existing controls stand: “Rules for crawler product tokens and proprietary policy tokens keep whatever meaning they have today.” The draft is not yet a standard, and OpenAI called its support a “voluntary effort.”

Meta’s Terms

Meta is not among the four. Andy Stone, its vice president of communications, wrote on X on Friday that “creating a common signal that conveys such preferences to multiple providers at once is an important step,” but did not say whether Meta would honor it.

The same day, Britain’s Competition and Markets Authority opened an investigation into whether Meta is abusing a dominant position with terms that could let it collect content from the websites of businesses and publishers that use its services, and use it for its own products, including its AI features, without giving them a choice.Chapter IIThe authority is acting under the Competition Act 1998, whose Chapter II prohibits abuse of a dominant position that may affect trade within the United Kingdom. Sarah Cardell, the authority’s chief executive, said that “where a company holds a dominant position, it must not use that position to impose unfair terms.”

Meta said there were “no grounds” for an investigation, Reuters reported. “We’ve already committed not to change our terms without consulting publishers and other stakeholders,” a Meta spokesperson said, according to Reuters. “We welcome the opportunity to explain our approach to the CMA.” Meta had put the terms off from Oct. 30 to Nov. 20, then on Oct. 28 announced a pause pending a consultation with industry groups. A Meta spokesperson had told Nieman Lab that content owners would still be able to use robots.txt to signal preferences; Nieman Lab reported that no version of the terms has included language about respecting them. The authority says it has not decided whether Meta broke the law.

The European Publishers Council has asked for more than a switch. “Publishers must retain control over additional uses of their content, including for AI, and the ability to negotiate licenses and payment,” its executive director, Angela Mills Wade, told Nieman Lab in an email about Meta’s terms, published Oct. 28. The signal arrives while the copyright suits over training go on: in the consolidated cases against OpenAI in New York, replies on the summary-judgment motions over fair use are due Nov. 20.

Daily Cortex is written with AI models, including Anthropic’s Claude, and fact-checked with OpenAI’s GPT models.