A robots.txt file is a plain text file at the root of your domain that tells crawlers which URLs they may request. The examples below cover the cases people search for most: allowing everything, disallowing all robots, blocking one folder, targeting a single bot, carving out exceptions, using wildcards and pointing to your sitemap. Each one comes with what it does and where it trips people up.
TL;DR
- To disallow all robots, use
User-agent: *followed byDisallow: /. To allow everything, leaveDisallow:empty or publish no file at all. - The file must be named exactly
robots.txt, be UTF-8 plain text, and sit at the root of the domain. - Robots.txt controls crawling, not indexing, and it is not security. A blocked URL can still show up in search if other pages link to it.
- When rules overlap, the most specific match wins, so a longer
Allowcan override a shorterDisallow. - Test every change before it goes live, especially on staging-to-production moves.
Table of Contents
- How a robots.txt file is built
- Example 1: Allow all robots to crawl everything
- Example 2: Disallow all robots
- Example 3: Block a folder or a single path
- Example 4: Block one crawler and leave the rest alone
- Example 5: Block a folder but allow one file inside it
- Example 6: Wildcards and end-of-URL matching
- Example 7: Add your sitemap
- Example 8: Rules for AI crawlers
- What robots.txt does not do
- Testing your file before it ships
- Common mistakes worth scanning for
- Our perspective: treat robots.txt as a monitored file
- Keep an eye on the file with Pingfloat
- A starter template you can adapt
- FAQ
- Sources
How a robots.txt file is built
A robots.txt file is made of one or more groups. Each group starts with a User-agent line that names the crawler, followed by Disallow or Allow lines that list paths. According to Google's documentation on creating a robots.txt file, the file has to be a plain, UTF-8 encoded text file named robots.txt and placed in the root directory of the site's domain.
That last part matters. A file at example.com/blog/robots.txt does nothing. The file for https://www.example.com/ lives at https://www.example.com/robots.txt.
Here is the anatomy of a small, realistic file:
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
Read it top to bottom:
User-agent: *opens a group that applies to every crawler. The asterisk matches all crawlers except the AdsBot family, which must be named explicitly.- Each
Disallowline gives a path relative to the domain root. Anything not disallowed is implicitly allowed. - The optional
Sitemapline gives the fully qualified URL of your sitemap, and it is not tied to any one group.
Lines starting with # are comments, which are handy for explaining to your future self why a rule exists.
Example 1: Allow all robots to crawl everything
User-agent: *
Disallow:
An empty Disallow value blocks nothing. Google's list of useful robots.txt rules notes that you can get the same result by omitting the robots.txt file entirely.
So why publish this file at all? Mostly for clarity and because it gives you a place to add the Sitemap line later. One practical benefit: crawlers requesting /robots.txt get a normal response instead of an error, which keeps your logs and monitoring cleaner.
Example 2: Disallow all robots
This is the rule behind the query "disallow all robots", and it is only two lines:
User-agent: *
Disallow: /
The slash is the root of the site, so every URL falls under the rule. Google describes this as disallowing crawling of the entire site for the matching crawlers. Two details to know:
- AdsBot is not covered. The wildcard does not match AdsBot crawlers, so if you want those blocked too, name them in their own group.
- It blocks crawling, not indexing. If other sites link to your pages, the bare URLs can still appear in results, just without a crawled description.
When is this the right rule? Typically on a staging site, a private preview or a site that is not ready for launch. When is it a disaster? When it ships to production by accident. A staging rule copied into a live deployment is one of the classic causes of a sudden traffic drop. We wrote a staging checklist for robots.txt disallow rules that walks through how to keep that from happening.
If you need a staging site to stay out of search results reliably, do not rely on robots.txt alone. Put it behind authentication, which is the only way to actually keep people and bots out.
Example 3: Block a folder or a single path
User-agent: *
Disallow: /private-folder/
Disallow: /cart/
Disallow: /checkout/
Paths are relative to the root and matching is by prefix. Disallow: /cart/ covers /cart/, /cart/items and anything else beneath it. Two common slips:
- Missing trailing slash.
Disallow: /cartalso blocks/cart-tips.htmland/carts, because it matches any path that begins with those characters. Add the slash when you mean a folder. - Case sensitivity. Paths are matched literally, so
/Private/and/private/are different folders. Write the path exactly as it appears in your URLs.
Typical candidates for blocking are internal search results, cart and checkout steps, filtered listing pages that produce endless URL combinations, and admin areas. Each of those wastes crawl effort without giving searchers anything to land on.
Example 4: Block one crawler and leave the rest alone
You can name a specific bot in the User-agent line. This blocks only Googlebot:
User-agent: Googlebot
Disallow: /
User-agent: *
Disallow:
And this blocks only Bingbot from one folder:
User-agent: Bingbot
Disallow: /drafts/
A crawler follows the group that best matches its own name and ignores the others. That is why the second example does not need a catch-all group: everyone else has no rules, so everything is crawlable.
The catch is that a bot-specific group replaces the wildcard group for that bot. It does not add to it. If your * group blocks /cart/ and you then add a Googlebot group with one rule, Googlebot no longer sees the /cart/ rule unless you repeat it in its own group. Many broken files come from exactly this.
Not sure what string a given bot uses? Our reference on identifying Googlebot, Bingbot and AI crawlers lists the user agent tokens you will want to match, and the broader user agent strings reference covers the rest.
Example 5: Block a folder but allow one file inside it
User-agent: *
Disallow: /media/
Allow: /media/press-kit.pdf
The Allow directive carves out an exception. Which rule wins when both match? RFC 9309, the formal Robots Exclusion Protocol standard, says the most specific match applies, meaning the rule with the most octets (in plain terms, the longest matching path). /media/press-kit.pdf is longer than /media/, so the file stays crawlable while the rest of the folder is blocked.
This is also why the order of lines inside a group is not the thing to rely on. Specificity decides. If you write two rules of equal length that conflict, avoid depending on behavior that might vary between crawlers; rewrite them so one is clearly more specific.
Example 6: Wildcards and end-of-URL matching
Google supports two special characters in paths: * matches any sequence of characters, and $ marks the end of a URL. The useful rules page shows a pattern like /*.xls$ to target a file type. Some practical variations:
User-agent: *
# Block all PDF files
Disallow: /*.pdf$
# Block URLs with a session parameter
Disallow: /*?sessionid=
# Block a folder at any depth
Disallow: /*/tmp/
Without the $, /*.pdf would also match /guide.pdf?download=1 and /guide.pdf.html. With it, only URLs that end in .pdf match. Use wildcards sparingly. Every pattern is another thing that can match a URL you did not intend, so read each one against a handful of real URLs from your site before you publish.
Example 7: Add your sitemap
User-agent: *
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
The Sitemap line is optional and takes a full, absolute URL, protocol included. You can list more than one sitemap, one per line. It pairs naturally with your blocking rules: you tell crawlers what to skip and where the pages worth finding are.
If your sitemap is large, mind its limits and structure. We cover those in our guide to sitemap best practices and the 50,000-URL limit. A sitemap that lists URLs your robots.txt blocks sends crawlers mixed signals, so keep the two consistent.
Example 8: Rules for AI crawlers
Many site owners now want to treat AI crawlers differently from search crawlers. The mechanism is the same: name the bot and set the rule. A sketch:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /cart/
Two cautions. First, the token has to match what the operator documents, so check the operator's own page for the current name before you copy anything. Second, robots.txt is a convention that well-behaved crawlers follow. It is a request, not a lock, which brings us to the limits.
What robots.txt does not do
Three misunderstandings cause most of the pain.
It does not block indexing. Google states that a page blocked by robots.txt can still appear in search results if it is linked from elsewhere. To keep a page out of results, you need a noindex rule that crawlers can actually read, which means the page must not be blocked in robots.txt. If you suspect a stray directive, our developer checklist on fixing unexpected noindex covers where it tends to hide, and the Meta Tag Checker lets you inspect a page's robots meta tag quickly.
It is not access control. Google's guidance is that robots.txt should not be used to secure private data, and private content belongs behind authentication. There is a second problem: robots.txt is public. Anyone can open it, so a line like Disallow: /secret-admin/ publishes the folder name to the world.
It is not a precise tool for removal. Blocking a URL after it is already indexed does not remove it. It only stops crawlers from refreshing what they know about it.
Testing your file before it ships
Treat robots.txt like code. A single misplaced character can hide a whole site. A reasonable routine:
- Check the location. Open
https://yourdomain.com/robots.txtin a browser. It should load as plain text, not redirect to an HTML page. - Run it through a validator. A robots.txt validator and checker catches syntax errors and shows how directives are read.
- Use Google's own tools. Google says you can test an accessible file with the robots.txt report in Search Console, or test locally with its open-source robots.txt library.
- Spot-check real URLs. Pick ten URLs you want crawled and ten you want blocked, and confirm each behaves as expected.
- Watch the site after deploy. A rule that accidentally blocks a section will not announce itself.
If Search Console reports a URL as blocked by robots.txt and you did not mean it, find the matching Disallow line, remember that the most specific rule wins, and either narrow the pattern or add a more specific Allow. Then re-test the exact URL.
Common mistakes worth scanning for
| Mistake | What happens | Fix |
|---|---|---|
Disallow: / left over from staging | Whole site blocked from crawling | Review robots.txt in every release checklist |
| File placed in a subfolder | Crawlers never read it | Put it at the domain root |
| Bot-specific group without shared rules | That bot ignores your * rules | Repeat shared rules in each group |
| Blocking a page you want deindexed | Crawlers cannot see the noindex tag | Allow crawling and use noindex |
| Listing private folders | Publicly reveals their names | Use authentication instead |
| Wrong encoding or filename | File may be ignored or misread | Save as UTF-8 plain text named robots.txt |
Our perspective: treat robots.txt as a monitored file
We build developer tools for checking the things that quietly break websites, and robots.txt is high on that list. It is tiny, it rarely changes, and so nobody looks at it until something goes wrong. Then the failure is large: an entire site vanishing from crawling because of a two-line file.
Our view is that the file deserves the same care as any production config. Keep it in version control. Review it in the same pull request as the deploy that changes it. Keep the staging rule and the production rule in separate files so one cannot overwrite the other. And check the live URL after each release, not just the repository.
We also think people overuse it. Reach for robots.txt to manage crawling of low-value URL sets, and reach for noindex, canonical tags or authentication for everything else. The simplest file that does the job is usually the right one.
— Andreas
Keep an eye on the file with Pingfloat
A robots.txt rule only helps if the site behind it is up and reachable. Crawlers that hit repeated server errors will not crawl well, and the 500 Internal Server Error and 503 Service Unavailable references explain what those responses mean. Pingfloat's free Website Uptime Monitor watches your site for availability problems, and the Meta Tag Checker and HTML Validator help you inspect the pages your rules are meant to protect or expose. You can browse the whole set from the Pingfloat home page.
A starter template you can adapt
For a typical public site with a store and internal search, this is a sensible place to begin:
# Rules for all crawlers
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Disallow: /*?sessionid=
# Sitemap
Sitemap: https://www.example.com/sitemap.xml
Swap in your own paths, then run through the testing steps above. If you are unsure whether a rule is needed, leave it out. Missing rules can be added in minutes, while an overbroad one can cost weeks of visibility before you notice.
FAQ
What is the structure of a robots.txt file?
It is a plain UTF-8 text file made of groups. Each group begins with a User-agent line and is followed by Disallow and Allow lines that name paths. You can add an optional Sitemap line with the full URL of your sitemap.
Can robots.txt block indexing?
No. It controls crawling, and Google notes that a blocked page can still appear in results if other pages link to it. To keep a page out of search, use a noindex directive that crawlers can read, or protect the page with authentication.
How do I fix a "blocked by robots.txt" error?
Find the Disallow rule that matches the URL, then narrow it or add a more specific Allow rule, since the most specific match wins. Test the exact URL with the Search Console robots.txt report afterward. If the block came from a staging file, replace it with the production version.
Where do I find a robots.txt file?
It always lives at the root of the domain, for example https://www.example.com/robots.txt. Type that address into a browser to read any site's file. If you get a not-found page, the site has no robots.txt, which generally means crawlers may access everything.
Sources
- Create and Submit a robots.txt File, Google Crawling Infrastructure
- Useful robots.txt Rules, Google Crawling Infrastructure
- Robots.txt Introduction and Guide, Google Search Central
- RFC 9309: Robots Exclusion Protocol, RFC Editor