Use Disallow: / to block crawling of an entire site, Disallow: /folder/ for a directory, Disallow: /page.html for a single page, and Disallow: /*.gif$ to block a file type. Combine these with Allow: overrides for exceptions inside a blocked folder. One caveat matters more than any syntax detail: Disallow stops crawling, not necessarily indexing, so pages blocked this way can still show up in search results if something else links to them. The copy-paste examples below cover each pattern, with notes on when to reach for it.
TL;DR:
- Disallow rules are evaluated based on the longest matching path, with specific Allow directives taking precedence over broader Disallow entries.
- Robots.txt should not be relied on for sensitive data protection, as it is publicly accessible and can reveal site structure.
- Blocking CSS or JavaScript files may hinder search engine rendering, so keep these assets accessible even if pages are restricted.
- To prevent indexing, pair Disallow with noindex tags or headers, since blocked pages can still appear in search results from external links.
- Test all changes thoroughly in staging, verify bot identification, and monitor site performance after deployment to avoid unintended visibility issues.
Table of Contents
- Copy-Paste Robots.txt Disallow Examples by Use Case
- How Crawlers Decide Which Rule Wins
- Common Pitfalls and Best Practices When Using Disallow
- Testing, Staging and Validation Checklist
- Blocking Specific Crawlers or Bots by Name
- Combining Multiple User-Agent Blocks in One File
- Syntax Mistakes That Quietly Break Robots.txt
- A Test-First Approach to Robots.txt Changes
- Pingfloat Tools for Validating and Monitoring Robots.txt Changes
- FAQ
- Sources
Copy-Paste Robots.txt Disallow Examples by Use Case
Each block below is a complete, working group. Swap in your own paths and test before deploying.
Block the entire site (useful for staging servers or pre-launch environments):
User-agent: *
Disallow: /
Allow everything except a few paths (the more common real-world need):
User-agent: *
Disallow: /admin/
Disallow: /cart/
Block a single directory, such as internal search results or a staging subfolder:
User-agent: *
Disallow: /private/
Leaving off the trailing slash changes the meaning: Disallow: /private blocks /private, /private.html, and /private-notes/, not just the folder. The slash matters.
Block one specific page:
User-agent: *
Disallow: /secret.html
Block file types with wildcards and the end-of-string anchor, handy for stopping crawlers from fetching large media or export files:
User-agent: *
Disallow: /*.gif$
Disallow: /*.xls$
The $ anchors the match to the end of the URL, so /*.gif$ catches photo.gif but leaves photo.gif.html alone.
Allow an exception inside a disallowed folder, which is where most people get tripped up:
User-agent: *
Disallow: /private/
Allow: /private/public-allowed.html
Allowlist structure, blocking everyone except a named crawler:
- List the crawler you want to permit in its own group with no Disallow line, or an empty one.
- Follow it with a
User-agent: *group carryingDisallow: /to catch everyone else. - Keep the specific group above the wildcard group for readability, though order doesn't affect matching logic.
How Crawlers Decide Which Rule Wins
When multiple rules could apply to the same URL, crawlers don't read top to bottom. They pick the rule with the longest matching path, a principle RFC 9309 sets out explicitly. A Disallow: /private/ next to an Allow: /private/reports/ means the Allow wins for anything under /reports/, because its path is more specific.
When an Allow and a Disallow match a URL with equal specificity, RFC 9309 and Google's own guidance agree that the Allow rule takes precedence. User-agent groups add another layer: a crawler looks for the group written for its exact product token first, and only falls back to the wildcard User-agent: * group when no specific one exists. It never merges rules from both.
Paths are case-sensitive and should be percent-encoded consistently with how your server serves them. /Folder/ and /folder/ are different paths to a crawler, and a mismatch between an encoded path in robots.txt and the live URL can silently break a rule.
Common Pitfalls and Best Practices When Using Disallow
Robots.txt is a public file. Anyone can view it at /robots.txt, which means a Disallow line naming /internal-tools/ or /beta-dashboard/ effectively broadcasts that path exists. Security researchers have flagged this as a recurring exposure risk: never rely on Disallow to protect anything sensitive. Use authentication, not a crawler directive, for that job.
A frequent mistake is blocking the CSS or JavaScript a page needs to render. Search engines that can't fetch those files may misjudge the page's content or layout, so Google's spec documentation recommends leaving rendering assets open even when the pages that use them are restricted elsewhere.
If the goal is keeping a page out of search results rather than just slowing crawlers down, Disallow is the wrong tool on its own. A blocked page can still get indexed from external links, since the crawler never sees a noindex instruction it was never allowed to fetch. Guidance on the distinction between crawling and indexing recommends pairing Disallow with a noindex meta tag or X-Robots-Tag header, or using noindex alone when the page must still be crawlable.
Keep rules minimal, as SEO practitioners generally advise, document every change, and list your sitemap location near the top of the file so crawlers find it without guessing.
Testing, Staging and Validation Checklist
Treat a robots.txt edit the same way you'd treat a firewall rule change, not a typo fix.
- Draft the change in a staging copy of the file first, never directly on production.
- Simulate multiple crawlers against it, including Googlebot, Bingbot, and GPTBot, since a rule aimed at one bot can accidentally catch another.
- Run the file through a robots.txt syntax tester before anything goes live.
- Cross-check exact crawler strings against a user-agent reference so your
User-agent:lines match real product tokens. - If your crawl tools support it, simulate a crawl against staging to confirm the paths you meant to block actually got blocked and nothing else did.
- Keep the file in source control with a review step before deployment, and avoid pushing straight to production on a Friday afternoon.
- After deployment, watch your uptime monitoring for a stretch to catch any accidental sitewide block that slipped through.
Pro Tip: Change one group at a time and wait a few days before stacking the next edit, so if organic traffic drops you know exactly which line to blame.
Blocking Specific Crawlers or Bots by Name
Sometimes the goal isn't blocking everyone, just one crawler. Each bot gets its own User-agent: group with its own Disallow lines:
User-agent: Googlebot
Disallow: /example/
User-agent: *
Disallow:
This blocks Googlebot from /example/ while leaving every other crawler, including Bingbot and smaller search bots, free to crawl the same path. The empty Disallow: line in the wildcard group means "block nothing," which is a common way to write an explicit allow-all.
The reverse pattern, blocking one aggressive bot while leaving the rest of the site open to search engines, looks like this:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow:
Get the product token exactly right. Googlebot-Image and Googlebot are different tokens, and a rule written for one does not apply to the other unless you write both groups or deliberately rely on the wildcard fallback. When in doubt about what a specific crawler calls itself, checking a current list of crawler user-agent strings beats guessing from memory, since bots do rename themselves over time.
Combining Multiple User-Agent Blocks in One File
Most real-world robots.txt files aren't one group, they're several stacked together, each handling a different crawler or rule set:
User-agent: Googlebot
Disallow: /admin/
Disallow: /search/
User-agent: Bingbot
Disallow: /admin/
Disallow: /archive/
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Each group is evaluated independently. Googlebot reads only its own group and ignores Bingbot's rules entirely, and the wildcard group only applies to crawlers with no specific group of their own. A crawler never combines its own group with the wildcard group, it picks one.
A common error is assuming rules stack across groups, so a webmaster blocks /archive/ only in the Bingbot group and is surprised when Googlebot still crawls it. If a restriction should apply to every bot, it needs to appear in every relevant group, or just once in the wildcard group if no bot needs an exception. The sitemap directive can live anywhere in the file and applies globally regardless of which group it sits near.
Syntax Mistakes That Quietly Break Robots.txt
A few errors show up constantly, and each one tends to cause silent failures rather than obvious crashes.
Missing the colon after a directive, like Disallow /folder/ instead of Disallow: /folder/, makes the line invalid and most parsers will skip it. Forgetting the leading slash, writing Disallow: folder/ instead of Disallow: /folder/, means the rule never matches any real path, since every URL path starts with a slash.
Placing a directive outside any User-agent: group is another frequent problem. Rules written before the first User-agent: line, or dropped in after a blank line that breaks the group, are typically ignored. Practical implementations handling robots.txt at scale note that parsers generally discard stray directives that don't belong to a recognized group rather than guessing at intent.
Duplicate or conflicting Disallow lines for the same path inside one group waste space without changing behavior, since the longest match wins regardless of how many times a shorter rule repeats. And capitalizing paths inconsistently, writing /Private/ in robots.txt when the live URL is /private/, means the rule simply won't fire, because path matching is case-sensitive.
A Test-First Approach to Robots.txt Changes
I'd rather ship a boring robots.txt than a clever one. Every rule should earn its place, get tested against a few real crawler strings in staging, and go out one change at a time so a mistake is easy to spot and easy to reverse. Checking crawler identities and meta tags before and after a deployment catches more problems than any amount of careful reading of the file itself. Treat each edit as reversible until you've watched it behave correctly in production for a few days.
— Andreas
Pingfloat Tools for Validating and Monitoring Robots.txt Changes
Getting a Disallow rule right on paper is one thing, confirming it behaves correctly against real crawlers is another, and we built a few tools specifically for that gap.
Before trusting a User-agent: group, check it against a current reference: our User Agent Strings reference lists how major crawlers identify themselves, and What's My User Agent shows you the exact string a browser or script is currently sending. If your real goal is keeping a page out of search results rather than just slowing crawlers, our Meta Tag Checker confirms whether a noindex tag is actually present and readable, since that's the directive that controls indexing, not Disallow. Once a change ships, our Website Uptime Monitor flags unexpected downtime or misconfiguration before it costs you crawl budget or traffic. All three are free to use, no account required, so there's no reason to deploy a robots.txt change blind.
FAQ
Is Robots.txt Illegal?
No, robots.txt is a voluntary standard defined by RFC 9309, not a law. It tells well-behaved crawlers what you'd prefer they not access, but it carries no legal force and some crawlers ignore it entirely.
Can You Provide an Example of a Robots.txt File?
A minimal working file looks like this: User-agent: * followed by Disallow: /admin/ and a Sitemap: line pointing to your sitemap URL. This blocks every crawler from one folder while leaving the rest of the site open and pointing search engines to your sitemap.
How Do I Test User-Agent Rules Without Risking My Live Site?
Draft the change in a staging copy of robots.txt, then simulate the specific crawlers you care about against it using a syntax tester before touching production. Cross-reference the exact bot name against a current list of crawler user-agent strings so your User-agent: line matches the real product token.
Is Robots.txt Still Relevant?
Yes, it remains the standard way to guide crawler behavior, and RFC 9309 formalized it as an official protocol rather than just a convention. It still can't force indexing decisions on its own, which is why pairing it with noindex directives remains standard practice for anything that must stay out of search results.
Sources
- RFC 9309: Robots Exclusion Protocol
- How Google interprets the robots.txt specification | Google Developers
- Robots.txt guide — Search Engine Land
- Noindex vs Robots.txt — Rank Math
- Respecting Robots Exclusion Protocol at scale — Medium (GumGum tech)