Scraper.ioCoveragePublish /llms.txtllmsPublish /llms-full.txtfullRefused by robots.txtrefusedTotal
6 of 13 category buckets · counts follow the searchclick a category to filter
SiteCategoryllms.txtSizellms-fullChangedSrc
28 of 28 sites · ordered by Tranco rank · each row opens the /llms.txt it was probed at
Do something with this data
Connect your agent
MCP-readable — wire it straight into Claude, Codex or Cursor.
npx -y mcp-remote https://fwgzmlpzreapyvvtfmpo.supabase.co/functions/v1/mcp --header "Authorization:Bearer $SCRAPERIO_API_KEY"Set up MCP Get it as an API or CSV
Every column, every row, in the format your pipeline already reads.
Read the docsHave an agent run on it
Spin up a long-running agent on this feed — apply, alert, enrich. Built and run by scraper.io.
Ask for a managed agentAll rows, plain table
llms.txt Adoption Index: 400 records · updated 4 September 2026 · open — every row published here and in the CSV. which of the Tranco top 1,000 publish /llms.txt and /llms-full.txt, how big, and when they last changed
| # | Site | Category | llms.txt | Size (KB) | llms-full | Changed | Last changed | Source |
|---|---|---|---|---|---|---|---|---|
| 1 | google.com | search | no | — | no | — | google.com | |
| 2 | cloudflare.com | infrastructure | yes | 16.8 | yes | 2026-09-04 | cloudflare.com | |
| 3 | gstatic.com | infrastructure | robots-refused | — | robots-refused | — | gstatic.com | |
| 4 | facebook.com | social | robots-refused | — | robots-refused | — | facebook.com | |
| 5 | microsoft.com | other | no | — | no | 2025-09-18 | microsoft.com | |
| 6 | googleapis.com | infrastructure | no | — | no | — | googleapis.com | |
| 7 | amazonaws.com | infrastructure | no | — | no | — | amazonaws.com | |
| 8 | youtube.com | media | no | — | no | — | youtube.com | |
| 9 | akamai.net | infrastructure | unreachable | — | unreachable | — | akamai.net | |
| 10 | apple.com | other | no | — | no | — | apple.com | |
| 11 | instagram.com | social | robots-refused | — | robots-refused | — | instagram.com | |
| 12 | mail.ru | other | robots-refused | — | robots-refused | — | mail.ru | |
| 13 | ezviz7.com | other | unreachable | — | unreachable | — | ezviz7.com | |
| 14 | fbcdn.net | infrastructure | robots-refused | — | robots-refused | — | fbcdn.net | |
| 15 | dzen.ru | other | no | — | no | — | dzen.ru | |
| 16 | twitter.com | social | robots-refused | — | robots-refused | — | twitter.com | |
| 17 | linkedin.com | social | no | — | no | — | linkedin.com | |
| 18 | gtld-servers.net | other | unreachable | — | unreachable | — | gtld-servers.net | |
| 19 | domaincontrol.com | other | unreachable | — | unreachable | — | domaincontrol.com | |
| 20 | googlevideo.com | other | unreachable | — | unreachable | — | googlevideo.com | |
| 21 | office.com | other | no | — | no | — | office.com | |
| 22 | googletagmanager.com | adtech | no | — | no | — | googletagmanager.com | |
| 23 | hicloudcam.com | other | unreachable | — | unreachable | — | hicloudcam.com | |
| 24 | live.com | other | no | — | no | — | live.com | |
| 25 | akamaiedge.net | other | unreachable | — | unreachable | — | akamaiedge.net | |
| 26 | amazon.com | commerce | no | — | no | — | amazon.com | |
| 27 | akadns.net | other | unreachable | — | unreachable | — | akadns.net | |
| 28 | azure.com | other | yes | 45.9 | no | 2026-02-17 | azure.com | |
| 29 | bing.com | search | no | — | no | — | bing.com | |
| 30 | github.com | developer | yes | 28 | yes | 2026-09-04 | github.com | |
| 31 | wikipedia.org | reference | no | — | no | — | wikipedia.org | |
| 32 | whatsapp.net | other | no | — | no | — | whatsapp.net | |
| 33 | apple-dns.net | other | unreachable | — | unreachable | — | apple-dns.net | |
| 34 | googleusercontent.com | other | no | — | no | — | googleusercontent.com | |
| 35 | fastly.net | infrastructure | yes | 15.4 | no | 2026-09-04 | fastly.net | |
| 36 | appsflyersdk.com | other | robots-refused | — | robots-refused | — | appsflyersdk.com | |
| 37 | doubleclick.net | adtech | no | — | no | — | doubleclick.net | |
| 38 | aaplimg.com | infrastructure | unreachable | — | unreachable | — | aaplimg.com | |
| 39 | microsoftonline.com | other | unreachable | — | unreachable | — | microsoftonline.com | |
| 40 | office.net | other | unreachable | — | unreachable | — | office.net | |
| 41 | netflix.com | media | robots-refused | — | robots-refused | — | netflix.com | |
| 42 | trafficmanager.net | other | unreachable | — | unreachable | — | trafficmanager.net | |
| 43 | gandi.net | other | no | — | no | — | gandi.net | |
| 44 | sharepoint.com | other | unreachable | — | unreachable | — | sharepoint.com | |
| 45 | youtu.be | other | no | — | no | — | youtu.be | |
| 46 | digicert.com | other | yes | 8.2 | no | 2026-02-19 | digicert.com | |
| 47 | cloud.microsoft | other | no | — | no | — | cloud.microsoft | |
| 48 | wordpress.org | other | yes | 5.6 | no | 2026-09-04 | wordpress.org | |
| 49 | skype.com | other | no | — | no | — | skype.com | |
| 50 | x.com | social | robots-refused | — | robots-refused | — | x.com |
350 further records are in the downloads. Take the whole feed as CSV or JSON — open data, published under CC BY 4.0. Published by Scraper.io. These are the run’s rows as exported; the interactive view above applies each feed’s own quality filters on top of them.