A robots.txt file tells search engines and other crawlers which parts of your site they may visit. One wrong line can stop Google crawling your whole site, so it’s worth understanding exactly how it works. This guide covers the syntax Google supports, with examples you can copy, and you can build your own with the robots.txt generator.
Quick answer: robots.txt is a plain text file at the root of your domain, such as example.com/robots.txt. It uses User-agent, Disallow, Allow and Sitemap lines to manage which URLs crawlers can access. It doesn’t keep pages out of Google: a blocked URL can still be indexed. To keep a page out of search results, use a noindex tag and let it be crawled.
What robots.txt does, and doesn’t do
Google describes robots.txt as a way to manage crawler traffic to your site, mainly to avoid overloading your server or wasting crawls on unimportant pages. It’s important to know its limits:
- It isn’t a way to hide pages. If other pages link to a blocked URL, Google can still index it, and it may appear in results without a description.
- It isn’t security. The file is public, and well-behaved crawlers follow it voluntarily. Bad bots can ignore it. Protect private content with a password.
- It can’t remove a page that’s already indexed. Use a
noindextag instead, and don’t block the page in robots.txt, or Google will never see the tag.
Where the file goes
It must be at the top level of the host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored. Rules apply only to the exact host, protocol and port, so shop.example.com needs its own file. Google ignores anything past 500 KiB.
The rules Google supports
| Line | What it does | Example |
|---|---|---|
| User-agent | Which crawler the group of rules applies to. * means all crawlers | User-agent: * |
| Disallow | A path the crawler shouldn’t crawl | Disallow: /checkout/ |
| Allow | A path it may crawl, used to make exceptions | Allow: /checkout/help/ |
| Sitemap | The full URL of your XML sitemap | Sitemap: https://www.example.com/sitemap.xml |
Google doesn’t support crawl-delay, and it stopped following noindex in robots.txt in 2019. Some other crawlers do read crawl-delay.
Wildcards and which rule wins
*matches any sequence of characters.Disallow: /*?sort=blocks any URL containing?sort=.$marks the end of the URL.Disallow: /*.pdf$blocks URLs ending in .pdf.- Paths are case-sensitive:
/Admin/and/admin/are different. - When an Allow and a Disallow rule both match, Google follows the most specific one, meaning the longest path. If they’re equally long, it uses the less restrictive rule, which is Allow.
Examples you can copy
Allow everything (the same as having no file):
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
A typical shop or WordPress site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?orderby=
Sitemap: https://www.example.com/sitemap.xml
Block one crawler completely:
User-agent: GPTBot
Disallow: /
To choose which AI crawlers to block, use the AI crawler robots.txt generator.
Common mistakes
Disallow: /underUser-agent: *. This blocks your whole site. It’s often left over from a staging site.- Blocking CSS and JavaScript. Google needs them to render your pages properly.
- Blocking pages you want removed. Google can’t see a noindex tag on a page it isn’t allowed to crawl.
- Server errors. If robots.txt returns a 5xx error, Google stops crawling the site for a while. A 404 is treated as “no restrictions”.
Google usually caches your robots.txt for up to 24 hours, so changes aren’t instant. Test a URL against your rules with the robots.txt tester, and build a sitemap with the XML sitemap generator. More tools are in the SEO tools guide.
Based on Google’s robots.txt documentation and RFC 9309. Other search engines and crawlers may interpret some rules differently.
Frequently asked questions
What is robots.txt used for?
To manage which URLs search engines and other crawlers can access, mainly to avoid overloading your server or wasting crawls on unimportant pages.
Does robots.txt stop a page appearing in Google?
No. A blocked URL can still be indexed if other pages link to it. Use a noindex tag, and let the page be crawled, to keep it out of results.
Where should robots.txt be placed?
At the root of the host, such as example.com/robots.txt. Each subdomain needs its own file.
Does Google support crawl-delay?
No. Google ignores crawl-delay, although some other search engines use it.
What happens if I don't have a robots.txt file?
Crawlers assume they can crawl everything. A file that returns a 404 is treated the same way.
How long does Google take to see changes?
Google usually caches robots.txt for up to 24 hours, so changes can take a day to apply.
Sources
- Google Search Central: Introduction to robots.txt
- Google Search Central: How Google interprets the robots.txt specification
- RFC 9309: Robots Exclusion Protocol
Last updated: