About CrawlCheck
Most sites were configured before AI crawlers existed. A single Disallow: / can make you invisible across ChatGPT, Perplexity and Claude at once - and you would never know.
CrawlCheck is a free, browser-based GEO audit toolkit. Paste your robots.txt and it checks it against 60+ AI crawlers - GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, Bytespider, CCBot and more - showing exactly which are blocked and where a path-level rule cuts off your best pages.
It also audits the signals AI engines quote from: JSON-LD structured data, llms.txt, noindex tags and extractable text. Every issue comes with a copy-paste fix, plus a generator with presets for 'allow AI search only' and 'block training'.
Everything runs locally in your browser. Nothing is uploaded, logged or tracked.
Key features
- Audits robots.txt against 60+ AI crawlers at 8 path levels
- Live crawler fetch test - catches server-level blocks and 403s
- Generates AI-ready robots.txt, llms.txt and JSON-LD snippets