robots.txt Examples for Every Situation

No need to write your robots.txt from scratch. Grab the example that matches your situation, adjust the paths, and deploy. Every example below is ready to copy and paste.

1. Allow Everything

The simplest possible robots.txt. All crawlers can access all pages.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

When to use: New sites, personal blogs, or any site where you want maximum crawl coverage. If you have nothing to hide from crawlers, this is your starting point.

2. Block Everything

Blocks all crawlers from all pages. Nuclear option.

User-agent: *
Disallow: /

When to use: Staging environments, development servers, or pre-launch sites. Remove this immediately when you go live -- forgetting to remove it is one of the most common SEO disasters.

Double-check before deploying

A Disallow: / under User-agent: * will completely deindex your site from search engines. Google can begin removing pages from search results within hours. Only use this on sites that should never appear in search.

3. Block a Single Directory

Blocks all crawlers from one directory while allowing everything else.

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

When to use: Most sites have at least one directory that crawlers should skip -- admin panels, internal tools, or API endpoints. Add more Disallow lines for additional directories.

4. Block a Single Page

Blocks a specific page rather than an entire directory.

User-agent: *
Disallow: /thank-you
Disallow: /old-landing-page

Sitemap: https://example.com/sitemap.xml

When to use: When you have individual pages that provide no SEO value, like thank-you pages, confirmation pages, or deprecated landing pages. Note that Disallow: /thank-you also blocks /thank-you-2 and /thank-you/details -- add a $ anchor if you need an exact match: Disallow: /thank-you$.

5. WordPress Standard

A practical robots.txt for WordPress sites.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

When to use: Any standard WordPress site. The key detail is Allow: /wp-admin/admin-ajax.php -- WordPress themes and plugins rely on this endpoint for front-end functionality, and blocking it can break how Google renders your pages. The search URL patterns prevent crawlers from indexing your internal search results.

Test your WordPress robots.txt

Paste your robots.txt and verify that your WordPress rules are working correctly.

6. Shopify Standard

Shopify generates a default robots.txt, but here is what a well-configured one looks like.

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /account
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*sort_by*
Disallow: /*?*page=*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known

Sitemap: https://example.com/sitemap.xml

When to use: Shopify stores. The + and %2B patterns block tag combination pages that create duplicate content. The sort_by and page parameters prevent indexing of filtered and paginated product listing variants.

7. E-commerce Site

A general e-commerce robots.txt for non-Shopify platforms.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /wishlist/
Disallow: /compare/
Disallow: /search?
Disallow: /*?sort=*
Disallow: /*?filter=*
Disallow: /*?page=*
Disallow: /order-confirmation/
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-categories.xml

When to use: Custom e-commerce platforms (Magento, WooCommerce, custom builds). The key pattern is blocking URL parameters for sorting, filtering, and pagination -- these create near-duplicate pages that waste crawl budget.

8. SaaS App with Public Marketing Site

Separates the public marketing site from the authenticated application.

# Public marketing pages - allow everything
User-agent: *
Allow: /
Allow: /blog/
Allow: /pricing
Allow: /features/
Allow: /about
Allow: /contact

# Block the application
Disallow: /app/
Disallow: /dashboard/
Disallow: /api/
Disallow: /settings/
Disallow: /auth/
Disallow: /invite/
Disallow: /onboarding/

# Block internal assets
Disallow: /_next/data/
Disallow: /*.json$

Sitemap: https://example.com/sitemap.xml

When to use: SaaS products where marketing pages and the app live on the same domain. You want crawlers indexing your landing pages and blog but staying away from authenticated app routes.

9. Block AI Crawlers Only

Blocks AI training crawlers while allowing search engines full access.

User-agent: *
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Cohere-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

When to use: When you want full search engine visibility but don't want your content used for AI model training. The AI crawler landscape changes frequently -- check current lists of known AI crawlers and update as needed.

Check your AI crawler blocking

Test your robots.txt against known AI crawler user agents to verify your rules are working.

10. Block All Except Google

Allows only Googlebot while blocking everyone else.

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

Sitemap: https://example.com/sitemap.xml

When to use: Rarely. This blocks Bing, DuckDuckGo, and all other search engines. Only use this if you have a specific reason to limit crawling to Google only, such as server resource constraints or a deliberate SEO strategy focused exclusively on Google.

11. Multiple Sitemaps

For sites with multiple sitemaps (large sites, multi-language, mixed content types).

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-pages.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-images.xml
Sitemap: https://example.com/sitemap-en.xml
Sitemap: https://example.com/sitemap-fr.xml
Sitemap: https://example.com/sitemap-de.xml

When to use: Large sites with thousands of pages, multilingual sites, or sites with distinct content types that benefit from separate sitemaps. You can list as many Sitemap directives as you need.

12. Complex Site with Mixed Rules

A real-world example combining multiple rule groups, wildcards, and exceptions.

# Search engines - standard access
User-agent: Googlebot
Disallow: /admin/
Disallow: /api/
Disallow: /internal/
Allow: /api/public/
Disallow: /*?preview=true
Disallow: /*?utm_*

User-agent: Bingbot
Disallow: /admin/
Disallow: /api/
Disallow: /internal/
Allow: /api/public/
Crawl-delay: 5

# All other search engines
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /internal/
Disallow: /staging/
Disallow: /tmp/
Disallow: /*.json$
Disallow: /*?*

# Block AI crawlers entirely
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Sitemaps
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-blog.xml

When to use: Large or complex sites that need different rules for different crawlers. This example gives Google full query-parameter control, slows Bing down, gives generic crawlers stricter rules, and blocks AI crawlers entirely.

Quick Comparison

Scenario	Key Rule
Allow everything	Allow: /
Block everything	Disallow: /
Block one directory	Disallow: /path/
Block AI crawlers	Per-agent Disallow: /
WordPress	Allow admin-ajax.php
E-commerce	Block sort/filter params
SaaS app	Block /app/ and /dashboard/

Always test after pasting

Even when using a template, always test your robots.txt after customizing it. A misplaced wildcard or missing slash can block pages you need indexed. Validate the file and test specific URLs against your rules before deploying.

Copy. Paste. Test. Deploy.

Test your robots.txt for free

Validate your robots.txt file instantly. Check directives, find crawling issues, and ensure search engines can access your site.