A robots.txt file tells search engines and other crawlers which parts of your site they may visit. One wrong line can stop Google crawling your whole site, so it’s worth understanding exactly how it works. This guide covers the syntax Google supports, with examples you can copy, and you can build your own with the robots.txt generator.

Quick answer: robots.txt is a plain text file at the root of your domain, such as example.com/robots.txt. It uses User-agent, Disallow, Allow and Sitemap lines to manage which URLs crawlers can access. It doesn’t keep pages out of Google: a blocked URL can still be indexed. To keep a page out of search results, use a noindex tag and let it be crawled.

What robots.txt does, and doesn’t do

Google describes robots.txt as a way to manage crawler traffic to your site, mainly to avoid overloading your server or wasting crawls on unimportant pages. It’s important to know its limits:

Where the file goes

It must be at the top level of the host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored. Rules apply only to the exact host, protocol and port, so shop.example.com needs its own file. Google ignores anything past 500 KiB.

The rules Google supports

Line What it does Example
User-agent Which crawler the group of rules applies to. * means all crawlers User-agent: *
Disallow A path the crawler shouldn’t crawl Disallow: /checkout/
Allow A path it may crawl, used to make exceptions Allow: /checkout/help/
Sitemap The full URL of your XML sitemap Sitemap: https://www.example.com/sitemap.xml

Google doesn’t support crawl-delay, and it stopped following noindex in robots.txt in 2019. Some other crawlers do read crawl-delay.

Wildcards and which rule wins

Examples you can copy

Allow everything (the same as having no file):

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

A typical shop or WordPress site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?orderby=

Sitemap: https://www.example.com/sitemap.xml

Block one crawler completely:

User-agent: GPTBot
Disallow: /

To choose which AI crawlers to block, use the AI crawler robots.txt generator.

Common mistakes

Google usually caches your robots.txt for up to 24 hours, so changes aren’t instant. Test a URL against your rules with the robots.txt tester, and build a sitemap with the XML sitemap generator. More tools are in the SEO tools guide.

Based on Google’s robots.txt documentation and RFC 9309. Other search engines and crawlers may interpret some rules differently.

Frequently asked questions

What is robots.txt used for?

To manage which URLs search engines and other crawlers can access, mainly to avoid overloading your server or wasting crawls on unimportant pages.

Does robots.txt stop a page appearing in Google?

No. A blocked URL can still be indexed if other pages link to it. Use a noindex tag, and let the page be crawled, to keep it out of results.

Where should robots.txt be placed?

At the root of the host, such as example.com/robots.txt. Each subdomain needs its own file.

Does Google support crawl-delay?

No. Google ignores crawl-delay, although some other search engines use it.

What happens if I don't have a robots.txt file?

Crawlers assume they can crawl everything. A file that returns a 404 is treated the same way.

How long does Google take to see changes?

Google usually caches robots.txt for up to 24 hours, so changes can take a day to apply.

Sources

Last updated: