Create a correct robots.txt file for your website in a few clicks. Add the folders you want to keep crawlers out of, your sitemap, WordPress rules and, if you want, rules that block AI training crawlers.
Loading the tool…
What robots.txt does
robots.txt is a plain text file at the root of your site (example.com/robots.txt) that tells search engine crawlers which parts of the site they may visit. It’s a request, not a lock: well-behaved bots follow it, but it doesn’t hide pages from people.
Blocking a URL in robots.txt doesn’t remove it from Google. If other sites link to it, Google can still index the URL without its content. To keep a page out of search results, let it be crawled and add a noindex meta tag.
Examples
A typical online shop
Block the basket, checkout and internal search results, which create thousands of thin pages, and point to the sitemap:
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /search Sitemap: https://example.com/sitemap.xml
A staging site
Ticking “block everything” adds Disallow: /. Use it on test sites, but remember to remove it at launch: it’s one of the most common reasons a new site never appears in Google.
Blocking AI crawlers
Some AI companies use separate crawlers for training data, such as GPTBot and Google-Extended. Blocking them doesn’t affect normal Google Search rankings, but it may reduce how often your content is used or cited by AI tools. Decide based on whether you want that visibility.
Frequently asked questions
Where do I put robots.txt?
At the root of your domain, so it’s reachable at https://yourdomain.com/robots.txt. It won’t work in a subfolder.
Can robots.txt stop a page being indexed?
No. Use a noindex meta tag or HTTP header instead, and don’t block that page in robots.txt, or Google won’t see the tag.
Should I block AI crawlers?
It’s your choice. Blocking training crawlers doesn’t harm Google rankings, but it may reduce your visibility in AI answers.
Does WordPress create robots.txt automatically?
Yes, a virtual one. SEO plugins let you edit it, or you can upload a real file to override it.
Sources
Last updated: