Paste your robots.txt and check whether specific URLs are allowed or blocked for Googlebot or any other crawler.
Loading the tool…
How Google reads robots.txt
| Rule | What it means |
|---|---|
| Most specific group wins | Googlebot follows a Googlebot group if there is one, and ignores the * group |
| Longest match wins | Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ |
| Ties go to Allow | If an Allow and a Disallow rule are the same length, Allow wins |
| * wildcard | Matches any characters: Disallow: /*?s= blocks search result pages |
| $ end marker | Disallow: /*.pdf$ blocks URLs ending in .pdf |
| Case-sensitive | /Blog/ and /blog/ are different paths |
Blocked isn’t the same as removed
robots.txt stops crawling, not indexing. A blocked page can still appear in search results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex tag instead.
Since Search Console’s tester was retired
Google replaced its old robots.txt Tester with a robots.txt report in Search Console, which shows fetch status and errors. To test individual URLs against your rules, use a tool like this one or the URL Inspection tool.
Frequently asked questions
Why is my page blocked by robots.txt?
Check the matching rule shown: often a broad Disallow like /*? or a folder rule catches more than intended.
Does Googlebot obey crawl-delay?
No, Google ignores crawl-delay.
Is robots.txt case-sensitive?
Paths are; directive names like Disallow aren’t.
Can I test a robots.txt before uploading it?
Yes, paste the new version here and test your important URLs.
Sources
Last updated: