Sites Blocking AI Crawlers: What Is Left for Training Data?

Sites blocking AI crawlers are websites that tell the bots AI companies use to collect training data to stay away, usually with a rule in their robots.txt file and increasingly at the server itself. On 27 October 2026, 34 of the 151 most-visited domains that publish a robots.txt fully disallowed at least one of the main training crawlers, against 33 a month earlier on the same list. For a team that trains or fine-tunes models, that changes two things: which parts of the public web you can still use, and how you prove that you used them properly. This post covers what we measured, what changed in a month, why sites close the door to training while leaving it open to search, and how to collect fresh, regional data for a training pipeline without treating a site's "no" as a technical problem.

Which sites block AI crawlers?

The busiest sites are the most likely to block them, and they block training far more often than search. On 27 October 2026 we fetched robots.txt from the 300 most-visited domains on the Tranco list, from a home connection in Ireland, with a client that named itself as a survey. It was the same list, the same script and the same connection we used on 25 September for our post on how AI search fetches web pages. 151 domains served a robots.txt file. Many of the others are infrastructure domains with no website at all.

Of those 151, 34 fully disallow at least one of seven crawlers that collect pages for AI training or AI data use: OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, Google-Extended, Applebot-Extended, ByteDance's Bytespider and Meta's meta-externalagent. Nine refuse all seven: TikTok, MSN, Snapchat, The New York Times, SoundCloud, both BBC domains, Launchpad and Nature. Eleven refuse at least one training crawler while leaving every search and user fetcher in our list open.

The pattern is the one OpenAI's own crawler documentation invites. It says GPTBot "is used to make our generative AI foundation models more useful and safe", that OAI-SearchBot "is used to surface websites in search results", and that "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot". On our list, 16 domains do exactly that.

What changed in a month?

One large platform closed fully and one publisher reopened partly, so the count moved by one. Between 25 September and 27 October, facebook.com changed its rules for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended from a partial restriction to a full "Disallow: /". SoundCloud added full blocks for Anthropic's search and user fetchers, which makes it one of the nine sites that refuse everything we tested. CNN went the other way and removed its Google-Extended rule. Cisco removed its mentions of four training crawlers without restricting them. Every other domain that answered both times served the same rules for every crawler we track.

One month on 151 domains is a small sample, and we present it as a floor, not a trend line. It does show how the change happens: one large site at a time, without notice, in both directions. A source list that was checked once is already out of date a month later. We will run the same list again and publish the result beside this one.

Why do sites block AI training crawlers but allow search?

Because search sends visitors back and training does not. A site that appears in an AI answer can still get the click, the reader and the advertising impression. A site whose pages go into a model gets none of that, and loses control of how its work is reused. So site owners have started to answer the two questions separately, and the tools to do it have arrived.

The first tool is the per-crawler rule shown above. The second is a Content Signal, Cloudflare's line for robots.txt that states separately whether a site's content may be used for search, as input to an AI answer, or for "training or fine-tuning AI models": `ai-train=no, search=yes`. Six of the 151 domains publish one. Linktree and Launchpad say `ai-train=no`. Cloudflare, Sentry and Base say `ai-train=yes`. Forter publishes different signals for different crawlers.

The third tool is a default. Since 15 September 2026, new domains on Cloudflare block bots classified as Training or Agent on pages that display ads, while Search remains allowed. A site owner who never wrote a robots.txt rule can now be closed to training collection from the day the site goes live.

Is robots.txt the whole picture?

No. robots.txt is a request, not a lock, and the standard says so: RFC 9309 states that "these rules are not a form of access authorization". A crawler that ignores the file can still fetch the page. That is exactly why sites are moving their answer from the file to the server, and why a training team cannot treat "nothing in robots.txt" as "permission granted".

Three signs of that showed up in our own run. 15 of the 300 domains answered our request for the robots.txt file itself with a 403, so we could not read their rules at all. Cloudflare's default acts at the network edge, not in the file. And terms of service go further than robots.txt does: the Data Provenance Initiative's study Consent in Crisis audited the web domains behind the C4 training corpus and found that, in a single year from 2023 to 2024, new restrictions left more than 5% of all its tokens, and more than 28% of its most actively maintained, critical sources, fully restricted from use. Counting terms of service crawling restrictions, "a full 45% of C4 is now restricted".

For a training team, the practical reading is simple. The rules file tells you what a site has refused. It does not tell you what a site has agreed to. Those are two separate checks, and the second one is yours to make.

Where does fresh training data come from now?

From four places, and only one of them is the live web. Labs license content from publishers and platforms. They use archives collected before the rules changed. They generate synthetic data. And they still collect fresh public pages where the site allows it, because a model that only knows last year's web, in one language, from one country, is a weaker model.

That fourth source is the one that keeps getting harder to assemble, and it is where most of the risk sits. Fresh, regional coverage means requests from the markets you want to represent, at the time you need them, from pages whose rules permit the use you have in mind. Before a page enters a training set, run five checks.

1. What does robots.txt say to training crawlers?

Read the groups that name training crawlers as well as the group for every crawler. A site that disallows GPTBot and CCBot has told you what it thinks of training use, even if your own client has a different name.

2. Does the site publish a Content Signal?

If it says `ai-train=no`, the page does not go in the training set. If it says `ai-train=yes`, keep a copy of the line with the page.

3. What do the terms of service allow?

The Consent in Crisis figures show that terms restrict far more than robots.txt does. A page that is open to fetch can still be closed to reuse.

4. Where and when was the page requested?

Keep the market, the time of retrieval and the final address with every document. Training data with no origin record cannot be audited, cannot be refreshed and cannot be removed cleanly when a site changes its mind.

5. What came back if the page was refused?

Keep the refusal too. A 403 or a challenge page in a corpus is noise, and a refusal you recorded is evidence that your pipeline respected the answer.

How does Teneo fit a training pipeline?

Teneo supplies the fourth source: fresh public pages, requested from the market you name, through connections whose owners agreed to share them, paid per request. Bandwidth sharing changes where a request comes from. It does not change what a site has agreed to, and the five checks above apply to everything collected through Teneo as they do to anything else.

The connections come from Teneo Beacon, the app people install to share unused bandwidth from their own devices, with signed apps for Android, iOS, macOS, Windows and Linux. Beacon only shares bandwidth after the operator accepts a consent screen, and they can pause or quit at any time. It never accesses the operator's files or browsing, blocks connections to private and local-network addresses, and refuses ports used for abuse such as email sending and remote access. The traffic it carries comes from vetted business customers, not anonymous members of the public. Requests leave from real households and real mobile networks in the market being collected, which is what regional coverage needs. The Beacon page reported more than 8,300 Beacons online in October 2026.

The requests are made by agents. Hundreds of AI agents are deployed on the Teneo Protocol, and the data platforms in scope are X, Reddit, TikTok, LinkedIn, Instagram, YouTube, Google Maps, Google Search and Yelp. The market is part of the request, not a guess from the result: the Amazon agent's search command takes the storefront domain and currency as arguments. Several of those platforms appear in our survey's list of sites that refuse training crawlers, which is exactly why the checks belong to the buyer and why Teneo does not present any platform's data as cleared for training.

Every request is priced per command and paid in USDC through x402, the open standard that uses the HTTP 402 status code to ask for payment on the request itself. The Agent SDK documents the order of events: users see the price before executing a task, and payment is verified and settled on-chain before the agent processes it. Teneo's public agent skills list Base, Peaq and Avalanche as the supported networks. The Amazon agent's search and product commands are priced at 0.0025 dollars a query and its reviews at 0.001 dollars a query. A training team pays for the pages it actually collects, with no monthly minimum attached to a query.

Teneo does not publish a training-use filter, a provenance field set or a robots.txt check as part of an agent's response. If your pipeline needs those, and a training pipeline does, build them on your side and keep the record in the provenance card above beside every document.

Why is demand for fresh training data rising?

Because the share of the web that is open to training is shrinking while the appetite for current, regional data grows. The Consent in Crisis study measured the commons closing in a single year. Cloudflare's default since 15 September closes new ad-supported sites to training bots from their first day. And on our own fixed list, one of the largest platforms on the web moved from a partial rule to a full one in a month. Each step takes a source out of the easy pile.

At the same time, the other buyers in our guide to what bandwidth sharing is used for want the same thing for different reasons: AI search needs the live page per question, public data collection needs the page as served in each market, and ad verification needs proof from the market an ad was sold into. Fewer open doors and more teams that need what is behind the ones still open: that is why the supply that can make an honest, consented request from the right market, and leave a record of it, is worth more every year, and why bandwidth sharing is how it gets supplied.

Frequently asked questions

Which sites block AI crawlers?

On 27 October 2026, 34 of the 151 most-visited domains that publish a robots.txt fully disallowed at least one major AI training crawler. Nine refused all seven we tested: TikTok, MSN, Snapchat, The New York Times, SoundCloud, both BBC domains, Launchpad and Nature. OpenAI's and Anthropic's training crawlers are refused more than twice as often as their search fetchers.

Do sites block AI training crawlers but allow AI search?

Many do. OpenAI documents that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot, which is used for training. In our survey, 16 of 151 top domains fully disallow GPTBot without fully disallowing OAI-SearchBot, and 11 refuse at least one training crawler while leaving every search and user fetcher we tested open.

Is robots.txt legally binding?

The robots.txt standard, RFC 9309, says its rules "are not a form of access authorization". It is a public statement of what a site will and will not accept from automated clients, not a lock. Terms of service and local law can restrict reuse further, so a training team should treat robots.txt as one check among several, not as permission.

What is a Content Signal in robots.txt?

It is a line that states separately whether a site's content may be used for search, as live input to AI answers, or for AI training, for example "ai-train=no, search=yes". Six of the 151 top domains in our survey publish one; Linktree and Launchpad say ai-train=no, and Cloudflare, Sentry and Base say ai-train=yes.

Where do AI companies get training data if websites block crawlers?

From licensed content, archives collected earlier, synthetic data and fresh public pages that are still open to the use they intend. The last source is the one that keeps shrinking, so teams need requests from the right markets, a record of where and when each page was fetched, and a check of each site's rules before a page enters a training set.

Can Teneo collect pages from sites that block AI crawlers?

Teneo changes where a request comes from, not what a site has agreed to. Teneo does not publish a training-use filter, so the rules check is the buyer's: read robots.txt, Content Signals and terms for each source, keep the record with each document, and leave out pages a site has closed to training.

What to do next

If you train or fine-tune models on public web data, run the five checks on your current source list before your next collection, and keep the provenance record with every document. The survey script, the domain list, both months of results and the raw robots.txt files behind this post are published at teneo-protocol.ai/data/2026-10-27-robots-survey/, so you can run the same comparison on your own sources. If you want to supply the connections that fresh regional data needs, install Teneo Beacon and follow the setup guide for your platform. For the checklist that covers live web data more broadly, read how to get live web data for AI agents.