Robots.txt checker
Test a URL against your robots.txt for Google, Bing, the AI crawlers and any user agent you name, and find the lines that crawlers ignore or read differently.
Free and unlimited: does not use your weekly runsHow the tester reads your robots.txt
Like Google: a crawler follows the group with its own name, else the "User-agent: *" group. Within a group the longest matching rule wins, and Allow wins a tie with Disallow. "*" matches anything and "$" marks the end of the URL. Only the first 500 KiB of the file count.
For every crawler you see whether it may crawl the URL, the rule that decides it, and its Crawl-delay. The tool also lists the sitemaps the file names and the lines that crawlers ignore: misspelled or unknown directives, rules outside any group, full URLs where a path belongs, and noindex lines, which Google no longer reads.
Blocking is not hiding
Disallow stops crawling, not indexing: a blocked URL that other sites link to can still appear in Google, without a description. To keep a page out of search results, let it be crawled and give it a noindex tag.
Site audit: the full version
Crawl your site, get the issues sorted by impact and explained in plain language.
More free tools
Frequently asked questions
What if my site has no robots.txt?
Then every crawler may crawl everything, which is fine when you want nothing blocked. A robots.txt that answers 401, 403 or a server error is different: crawlers then stay away.
Can I test AI crawlers like GPTBot?
Yes, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are tested every time. The AI bot access checker also checks your firewall and page signals per bot.
Does this tool count towards my free runs?
No. It is free and unlimited, apart from a limit per site per hour.
Want the full picture?
Track your rankings, audit your site and see where AI answers mention you, with the full versions of these tools.
Start your free trial- 14-day free trial
- 25 keywords tracked daily
- No credit card