OtlumeAI Crawler Access Study 2026: SearchDock Reveals AI Crawler Access Data for Top 1,000 Websites
Key takeaway: SearchDock released the 'AI Crawler Access Study 2026,' which investigated the accessibility of AI crawlers across Tranco's Top 1,000 domains. The study found that 27.2% of websites blocked at least one AI crawler using robots.txt or server-side mechanisms. The report details the blocking ratios for major crawlers, including Bytespider, GPTBot, and ClaudeBot, and notes that 22.5% of websites published an llms.txt file.
- In the Tranco Top 1,000 domains scanned in 2026, 27.2% of sites blocked at least one AI crawler via robots.txt or server-side configuration.
- Bytespider had the highest blocking rate in robots.txt, representing 16.9% of the surveyed sites.
- 22.5% of accessible Top 1,000 sites published an llms.txt file intended for AI engines.
- CDNs like Cloudflare showed the highest blocking rate for GPTBot, preventing access for 11.2% of the 116 tested websites.
Otlume Editorial Team editorial team recommends this article on AI Crawlers in AI.
In the Tranco Top 1,000 domains scanned in 2026, 27.2% of sites blocked at least one AI crawler via robots.txt or server-side configuration.
—— Otlume Editorial Team · Editor's PickThis research provides quantified data and specific analyses, outlining the feasibility and barriers for large AI search engine crawlers to access website content. It offers practical guidance for B2B entities and developers aiming to have their content cited by AI.
Tags
This article is curated by the editorial team from public sources for reference only.