Free tools

Robots.txt and AI crawler checker

Check whether search engines and AI tools may crawl a page: your robots.txt rules for Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and more, plus noindex on the page itself.

Free, no sign-up. We read only public URLs and store nothing.

Search crawlers versus training crawlers

AI companies run separate crawlers. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages can appear in AI answers. Training crawlers such as GPTBot and ClaudeBot collect content for model training.

Blocking a training crawler doesn't remove you from AI search. Blocking a search crawler does. Many sites block the wrong one.

What we check

We read robots.txt with the same rules as the crawlers (most specific user-agent group, longest matching rule), test the page's path for each crawler, and check the page for noindex or nosnippet in its meta tags and X-Robots-Tag header. We also check that robots.txt lists a sitemap.

A robots.txt that can't be reached because of a server error is treated as "don't crawl anything", as crawlers do.

More free checks