Teneo robots.txt survey of the Tranco top 300, 25 September and 27 October 2026
Published with: https://teneo-protocol.ai/blog/ai-training-data-blocked-crawlers

What it is
- tranco-top300-2026-09-25.csv: the 300 most-visited domains on the Tranco list (https://tranco-list.eu/) as downloaded on 25 September 2026. The same list was used for both runs.
- robots-survey.mjs: fetches https://<domain>/robots.txt (then https://www.<domain>/robots.txt) once per domain, from a home broadband connection in Ireland, with a client that names itself as TeneoRobotsSurvey/1.0. For each AI crawler token it records whether a group naming the token fully disallows it ("Disallow: /"), partially restricts it, merely mentions it, or does not name it. The 27 October run also records Content-Signal lines and keeps each raw file.
- robots-survey-2026-09-25.json and robots-survey-2026-10-27.json: per-domain results of each run.
- compare-runs.mjs and compare-runs.json: the domain-by-domain comparison behind the post's figures.
- raw-2026-10-27/: the robots.txt files as served on 27 October 2026.

Limits
- One request per domain per run. robots.txt states a preference; it does not show what a server enforces, and 15 domains refused the request for the file itself with a 403 on 27 October.
- Many Tranco domains are infrastructure with no website; 151 served a robots.txt on 27 October and 152 on 25 September.
- "Training crawlers" in the post means GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent. "Search and user fetchers" means OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User.

Rerun: compare-runs.mjs reads robots-survey-2026-09-25.json and robots-survey.json. To check these figures, copy robots-survey-2026-10-27.json to robots-survey.json and run node compare-runs.mjs. To take a new reading, run node robots-survey.mjs 300 (it writes robots-survey.json), then node compare-runs.mjs.