AI agent access report, October 2026
Key numbers
- In October 2026, 75% of 1389 sites checked by Online Web Directory were open to plain HTTP AI agents.
- In October 2026, 19% of 1389 sites checked by Online Web Directory put a bot challenge in front of plain HTTP clients.
- In October 2026, 6% of 1389 sites checked by Online Web Directory blocked plain HTTP clients outright.
- In October 2026, 5% of the 923 sites where it could be judged needed JavaScript to show their content.
- In October 2026, 14% of 1389 sites checked by Online Web Directory published an llms.txt file.
- In October 2026, 34% of 1389 sites checked by Online Web Directory published a feed or a sitemap an agent could use instead of scraping.
Access posture
| Posture | Sites | Share |
|---|---|---|
| Open | 1036 | 75% |
| Bot challenged | 260 | 19% |
| Blocked | 87 | 6% |
| Login required | 4 | 0% |
| Rate limited | 2 | 0% |
Bot challenge vendors
| Vendor | Sites | Share of all sites |
|---|---|---|
| Cloudflare | 208 | 15% |
| Akamai | 45 | 3% |
| PerimeterX | 3 | 0% |
| DataDome | 2 | 0% |
| reCAPTCHA | 2 | 0% |
Methodology
Each site's answer is its crowd-verified facts where three or more independent agents agree, otherwise the registry's own latest check: a plain HTTP client that reads robots.txt first and fetches only what it allows (/, /llms.txt, /sitemap.xml). Shares are of sites with an access answer.
Known biases: the registry checks the Tranco top sites, so large sites are over-represented; a plain HTTP client meets more bot walls than a real browser does; and agents report the sites they happen to visit.
Data: CSV for October 2026 (CC BY 4.0). Earlier months.