robots.txt Allow Directive: How to Override Disallow Rules

The Disallow directive in robots.txt blocks crawlers from accessing URLs matching a path. But what if you want to block an entire directory except for one file inside it? That is where the Allow directive comes in. It lets you carve out exceptions within broader Disallow rules, giving you precise control over what gets crawled.

This guide covers how Allow works, how it interacts with Disallow when both match the same URL, practical patterns you will use regularly, and how different crawlers handle conflicts between the two directives. For the full robots.txt reference, see the robots.txt Guide.

What Allow Does

The Allow directive tells crawlers that a specific URL path is permitted for crawling, even if a Disallow rule would otherwise block it.

User-agent: *
Disallow: /private/
Allow: /private/public-page

In this example, every URL under /private/ is blocked except URLs that start with /private/public-page. A crawler requesting /private/internal-doc would be blocked. A crawler requesting /private/public-page would be allowed through.

Allow has been part of the robots.txt specification since it was formalized in RFC 9309. All major search engine crawlers support it, including Googlebot, Bingbot, and Yandex.

How Allow Interacts with Disallow

When a URL matches both an Allow rule and a Disallow rule, the crawler needs to decide which one takes priority. The answer depends on specificity: the rule with the longer (more specific) path prefix wins.

Specificity Wins

User-agent: *
Disallow: /images/
Allow: /images/public/

The URL /images/public/logo.png matches both rules. The Allow rule (/images/public/) is more specific (longer path) than the Disallow rule (/images/), so the Allow rule wins. The image is crawlable.

The URL /images/internal/secret.png only matches the Disallow rule. It is blocked.

Equal-Length Paths

When the Allow and Disallow paths are exactly the same length, the behavior is defined by RFC 9309: the Allow directive takes precedence. Google follows this convention.

User-agent: *
Disallow: /page
Allow: /page

Both rules match the URL /page and have the same path length. The Allow directive wins, so /page is crawlable.

In practice, you should avoid writing rules like this. Having an Allow and Disallow with the same path is confusing and serves no purpose. If you want a path to be crawlable, just do not Disallow it.

Order Does Not Matter

Unlike some configuration formats, the order of Allow and Disallow rules in robots.txt does not determine priority. Specificity determines priority. These two configurations are equivalent:

# Version A
User-agent: *
Disallow: /archive/
Allow: /archive/2025/

# Version B
User-agent: *
Allow: /archive/2025/
Disallow: /archive/

Both result in the same behavior: everything under /archive/ is blocked except URLs starting with /archive/2025/.

That said, putting Allow rules before or immediately after their corresponding Disallow rules makes the file easier to read. Readability matters when someone else needs to maintain the file six months from now. See our robots.txt syntax reference for formatting conventions.

Practical Examples

Allow One File in a Blocked Directory

The most common use case. You block a directory but need one specific file to remain crawlable, often for CSS, JavaScript, or a specific public resource.

User-agent: *
Disallow: /assets/
Allow: /assets/main.css
Allow: /assets/app.js

Crawlers can access main.css and app.js but nothing else in /assets/. This is useful when Google needs to render your pages (which requires CSS and JS) but you do not want other assets indexed.

Allow a Subdirectory in a Blocked Directory

User-agent: *
Disallow: /docs/
Allow: /docs/public/

Everything under /docs/ is blocked except the /docs/public/ subdirectory and everything within it. URLs like /docs/public/getting-started and /docs/public/api-reference are all crawlable.

Block Everything Except the Homepage

User-agent: *
Disallow: /
Allow: /$

The Disallow: / rule matches every URL on the site. The Allow: /$ rule uses the $ end-of-string marker to match only the exact root URL (/ and nothing after it). The result: only the homepage is crawlable. For more on pattern matching with $ and *, see robots.txt wildcards.

Allow Specific File Types in a Blocked Directory

Using wildcard patterns, you can allow specific file types:

User-agent: *
Disallow: /resources/
Allow: /resources/*.pdf$

This blocks everything under /resources/ except PDF files. The * matches any characters, and the $ ensures the URL ends with .pdf.

Allow a Specific Bot While Blocking Others

User-agent: *
Disallow: /api/

User-agent: Googlebot
Disallow: /api/
Allow: /api/public/

The general rule blocks /api/ for all crawlers. The Googlebot-specific block also blocks /api/, but with an Allow exception for /api/public/. Only Googlebot can access /api/public/. Every other crawler is blocked from the entire /api/ directory.

How Different Crawlers Handle Allow

All major crawlers follow the RFC 9309 specification for Allow and Disallow conflicts: the most specific (longest matching path) rule wins. But there are nuances.

Googlebot follows RFC 9309 strictly. Longest matching path wins. On equal length, Allow takes precedence. Google's documentation confirms this behavior explicitly.

Bingbot also follows the longest-match rule. Bing's behavior is consistent with Google's for Allow/Disallow resolution.

Yandex follows the same longest-match rule. Yandex was one of the early adopters of the Allow directive.

Less common crawlers may not support Allow at all. Older or simpler bots that only implement the original 1994 robots.txt convention may ignore Allow directives entirely and only process Disallow. If you are dealing with a specific crawler that ignores Allow, you may need to restructure your Disallow rules instead.

For blocking specific AI crawlers, the Allow directive works the same way. See how to block AI crawlers with robots.txt for bot-specific rules.

Common Patterns

E-commerce: Block Filtered URLs, Allow Category Pages

Online stores often have thousands of filtered URLs (sorted by price, color, size) that create duplicate content. You want to block the filtered versions but keep the main category pages.

User-agent: *
Disallow: /products/*?sort=
Disallow: /products/*?filter=
Allow: /products/

The Disallow rules block URLs with sort and filter query parameters. The Allow rule ensures the base product listing pages remain crawlable. The Disallow rules are more specific (longer path match) for URLs that contain those parameters, so they take priority for filtered URLs.

Blog: Block Tag and Author Archives, Allow Posts

User-agent: *
Disallow: /blog/tag/
Disallow: /blog/author/
Allow: /blog/

Tag and author archive pages are blocked (they often produce thin, duplicate content), but all blog posts under /blog/ remain accessible.

Staging Paths: Block Test Pages, Allow Specific Test Results

User-agent: *
Disallow: /staging/
Allow: /staging/approved/

Everything in staging is blocked except content that has been explicitly moved to the approved subdirectory.

Allow and the Disallow Directive

The Allow directive only makes sense in combination with Disallow. An Allow rule without a corresponding Disallow rule does nothing, because URLs are allowed by default. If no Disallow rule matches a URL, crawlers are free to access it regardless of whether an Allow rule exists.

# This Allow rule is pointless -- /blog/ is already allowed by default
User-agent: *
Allow: /blog/

You only need Allow when you are carving out exceptions to Disallow rules. For a full explanation of how Disallow works on its own, see robots.txt Disallow explained.

Testing Allow Rules

Getting Allow and Disallow interactions right is tricky. A single character difference in a path can change which rule matches. Always test your rules after writing them.

Test with a robots.txt tester. Paste your robots.txt content and test specific URLs to verify that the expected rule is winning. Pay special attention to edge cases: URLs at the boundary between allowed and disallowed paths.

Test with Google Search Console. Google Search Console has a robots.txt tester that shows you which rule matches a given URL and whether it is allowed or blocked. This is the definitive test for how Googlebot interprets your rules.

Test edge cases. For every Allow rule you write, test these scenarios:

• A URL that should match the Allow rule (should be crawlable).
• A URL that should match the Disallow rule (should be blocked).
• A URL that is right at the boundary (longest match should win).
• The directory path itself, with and without a trailing slash.

For a detailed testing process, see how to test your robots.txt.

Common Mistakes

Forgetting That Allow Needs a Matching Disallow

Writing Allow rules without corresponding Disallow rules clutters your robots.txt with rules that have no effect. Review your file periodically and remove orphaned Allow rules.

Using Allow to "Undo" a robots.txt Block for Indexing

If you blocked a URL with Disallow and Google indexed it anyway (showing "No information is available for this page"), adding an Allow rule restores crawling. But if the page was already deindexed via other means (noindex tag, URL removal tool), Allow does not re-index it. Allow only controls crawling, not indexing.

Overlapping Patterns with Wildcards

Wildcard patterns can create unexpected interactions:

User-agent: *
Disallow: /*.json$
Allow: /api/*.json$

This looks like it blocks all JSON files except those under /api/. And it does work that way for most crawlers. But the wildcard matching can behave subtly differently across crawlers. Test thoroughly when combining wildcards with Allow rules.

Assuming All Bots Support Allow

If you write rules that depend on Allow working correctly, and a bot that does not support Allow crawls your site, that bot will only see the Disallow rules. It will be blocked from paths you intended to allow. For critical crawlers, verify that they support the Allow directive.

References

1. RFC 9309 - Robots Exclusion Protocol
2. Google's robots.txt specification
3. Google Search Central - robots.txt introduction