Name: AI Crawler Access Index
Creator: Scraper.io
Published: baseline
License: https://creativecommons.org/licenses/by/4.0/

Scraper.io

Connect your agent

30 sites··weekly·next run Mondays 04:20 UTC

CoverageGPTBot blockedGPTClaudeBot blockedClaudePerplexity blockedPplxCCBot blockedCCTotal

4 of 4 stance buckets · counts follow the searchclick a stance to filter

SiteGPTBotClaudeBotPerplexityCCBotChangedSrc

30 of 30 sites · ordered by Tranco rank · each row opens the robots.txt it was read from

Do something with this data

Connect your agent

MCP-readable — wire it straight into Claude, Codex or Cursor.

npx -y mcp-remote https://fwgzmlpzreapyvvtfmpo.supabase.co/functions/v1/mcp --header "Authorization:Bearer $SCRAPERIO_API_KEY"

Set up MCP

Get it as an API or CSV

Every column, every row, in the format your pipeline already reads.

Read the docs

Have an agent run on it

Spin up a long-running agent on this feed — apply, alert, enrich. Built and run by scraper.io.

Ask for a managed agent

All rows, plain table

AI Crawler Access Index: 400 records · updated baseline · open — every row published here and in the CSV. robots.txt verdicts for nine AI crawlers across the Tranco top 1,000, with the directive line behind each

Last updated baseline · run 2026-09-04 · 50 of 400 rows shown

AI Crawler Access Index — 50 of 400 site records as of baseline. Every row links to the source it was read from.
#	Site	GPTBot	ClaudeBot	Perplexity	CCBot	Changed	Last changed	Source
1	google.com	allowed	allowed	allowed	allowed	baseline	baseline	google.com
2	cloudflare.com	allowed	allowed	allowed	allowed	baseline	baseline	cloudflare.com
3	gstatic.com	blocked	blocked	blocked	blocked	baseline	baseline	gstatic.com
4	facebook.com	allowed	allowed	allowed	blocked	baseline	baseline	facebook.com
5	microsoft.com	allowed	allowed	allowed	allowed	baseline	baseline	microsoft.com
6	googleapis.com	no robots	no robots	no robots	no robots	baseline	baseline	googleapis.com
7	amazonaws.com	allowed	allowed	allowed	allowed	baseline	baseline	amazonaws.com
8	youtube.com	allowed	allowed	allowed	allowed	baseline	baseline	youtube.com
9	akamai.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	akamai.net
10	apple.com	allowed	allowed	allowed	allowed	baseline	baseline	apple.com
11	instagram.com	blocked	blocked	blocked	blocked	baseline	baseline	instagram.com
12	mail.ru	allowed	allowed	allowed	allowed	baseline	baseline	mail.ru
13	ezviz7.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	ezviz7.com
14	fbcdn.net	blocked	blocked	blocked	blocked	baseline	baseline	fbcdn.net
15	dzen.ru	allowed	allowed	allowed	allowed	baseline	baseline	dzen.ru
16	twitter.com	blocked	blocked	blocked	blocked	baseline	baseline	twitter.com
17	linkedin.com	no robots	no robots	no robots	no robots	baseline	baseline	linkedin.com
18	gtld-servers.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	gtld-servers.net
19	domaincontrol.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	domaincontrol.com
20	googlevideo.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	googlevideo.com
21	office.com	no robots	no robots	no robots	no robots	baseline	baseline	office.com
22	googletagmanager.com	no robots	no robots	no robots	no robots	baseline	baseline	googletagmanager.com
23	hicloudcam.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	hicloudcam.com
24	live.com	no robots	no robots	no robots	no robots	baseline	baseline	live.com
25	akamaiedge.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	akamaiedge.net
26	amazon.com	blocked	blocked	blocked	blocked	baseline	baseline	amazon.com
27	akadns.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	akadns.net
28	azure.com	no robots	no robots	no robots	no robots	baseline	baseline	azure.com
29	bing.com	allowed	allowed	allowed	allowed	baseline	baseline	bing.com
30	github.com	allowed	allowed	allowed	allowed	baseline	baseline	github.com
31	wikipedia.org	allowed	allowed	allowed	allowed	baseline	baseline	wikipedia.org
32	whatsapp.net	blocked	blocked	blocked	allowed	baseline	baseline	whatsapp.net
33	apple-dns.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	apple-dns.net
34	googleusercontent.com	no robots	no robots	no robots	no robots	baseline	baseline	googleusercontent.com
35	fastly.net	allowed	allowed	allowed	allowed	baseline	baseline	fastly.net
36	appsflyersdk.com	blocked	blocked	blocked	blocked	baseline	baseline	appsflyersdk.com
37	doubleclick.net	allowed	allowed	allowed	allowed	baseline	baseline	doubleclick.net
38	aaplimg.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	aaplimg.com
39	microsoftonline.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	microsoftonline.com
40	office.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	office.net
41	netflix.com	allowed	allowed	allowed	blocked	baseline	baseline	netflix.com
42	trafficmanager.net	unreachable	unreachable	unreachable	unreachable	baseline	baseline	trafficmanager.net
43	gandi.net	allowed	allowed	allowed	allowed	baseline	baseline	gandi.net
44	sharepoint.com	unreachable	unreachable	unreachable	unreachable	baseline	baseline	sharepoint.com
45	youtu.be	allowed	allowed	allowed	allowed	baseline	baseline	youtu.be
46	digicert.com	allowed	allowed	allowed	allowed	baseline	baseline	digicert.com
47	cloud.microsoft	no robots	no robots	no robots	no robots	baseline	baseline	cloud.microsoft
48	wordpress.org	allowed	allowed	allowed	allowed	baseline	baseline	wordpress.org
49	skype.com	no robots	no robots	no robots	no robots	baseline	baseline	skype.com
50	x.com	blocked	blocked	blocked	blocked	baseline	baseline	x.com

350 further records are in the downloads. Take the whole feed as CSV or JSON — open data, published under CC BY 4.0. Published by Scraper.io. These are the run’s rows as exported; the interactive view above applies each feed’s own quality filters on top of them.