How to Fix "Blocked by robots.txt" Errors

You open Google Search Console and see pages marked as "Blocked by robots.txt." Your content is invisible to search engines, and you are losing organic traffic. The fix is usually straightforward once you understand what is happening.

This guide walks you through diagnosing the problem, identifying the offending rule, fixing it, and getting your pages re-crawled.

What "Blocked by robots.txt" Actually Means

When Google reports a page as "Blocked by robots.txt," it means Googlebot tried to crawl that URL but your robots.txt file told it not to. Googlebot respected the directive and did not fetch the page content.

There are two related but different statuses you might see in Google Search Console:

"Blocked by robots.txt" -- Google found the URL (through a sitemap, internal links, or external links) but cannot crawl it because of a Disallow rule. The page is not indexed.

"Indexed, though blocked by robots.txt" -- Google found the URL through links, and indexed the URL itself (showing it in search results), but could not crawl the page content. The search result will show the URL with no snippet or a snippet pulled from anchor text. This is the worst of both worlds: you are in search results but with no useful content shown.

Finding Blocked Pages in Google Search Console

If the blocked pages are ones you intentionally want hidden, no fix is needed. If they are pages you want in search results, you have a problem to solve.

Common Causes

Here are the most frequent reasons pages get blocked unintentionally.

Overly Broad Disallow Rules

The most common cause. A rule like Disallow: / blocks your entire site. A rule like Disallow: /blog blocks /blog, /blog/, /blog-post, and /bloggers -- anything starting with /blog.

# This blocks MORE than you probably intended
User-agent: *
Disallow: /blog

# This blocks only the /blog/ directory
User-agent: *
Disallow: /blog/

Wildcard Rules Gone Wrong

Wildcards are powerful but dangerous. A few common mistakes:

# Blocks every URL with a query string -- including paginated pages
Disallow: /*?

# Blocks every URL containing "page" anywhere in the path
Disallow: /*page*

# Blocks all .html files, even your key landing pages
Disallow: /*.html$

Blocking CSS, JavaScript, and Images

If you block resource directories, Googlebot cannot render your pages. Google treats this as a significant issue.

# Problematic rules that break rendering
Disallow: /wp-content/
Disallow: /static/
Disallow: /assets/css/
Disallow: /js/

CMS or Plugin Changes

WordPress plugins like Yoast SEO, Rank Math, or All in One SEO can modify your robots.txt. A plugin update or settings change can introduce new rules you did not expect. Similarly, Shopify's default robots.txt blocks certain paths that you might not realize are affected.

Diagnosing Which Rule Is Blocking

You know a page is blocked. Now you need to find which rule is causing it.

Step 1: Get your robots.txt. Open https://yourdomain.com/robots.txt in your browser.

Step 2: Identify the relevant User-agent block. For Google indexing issues, look for User-agent: Googlebot first. If there is no Googlebot-specific block, the User-agent: * block applies.

Step 3: Test the URL against each rule. For each Disallow and Allow rule in the block, check whether the blocked URL path matches the pattern.

For example, if the blocked URL is /blog/2024/my-great-post and your rules are:

User-agent: *
Disallow: /blog/2024/drafts/
Disallow: /blog/tag/
Disallow: /blog/*/comments

The third rule (/blog/*/comments) does not match. The first two do not match either. So the block must be coming from somewhere else -- maybe a different User-agent block or a rule you overlooked.

This manual process is tedious with complex files. A robots.txt testing tool does this instantly. Paste the file, enter the URL and user agent, and it tells you exactly which rule is matching.

How to Fix It

Once you know which rule is blocking your pages, you have three options.

Option 1: Remove the Rule

If the Disallow rule is not needed at all, remove it:

# Before
User-agent: *
Disallow: /blog/
Disallow: /admin/

# After -- removed the /blog/ rule
User-agent: *
Disallow: /admin/

Option 2: Make the Rule More Specific

If the rule is too broad, narrow it down:

# Before -- blocks all of /blog/
User-agent: *
Disallow: /blog/

# After -- only blocks drafts and tag pages
User-agent: *
Disallow: /blog/drafts/
Disallow: /blog/tag/

Option 3: Add an Allow Override

If you need the broad rule but want to exempt specific URLs:

User-agent: *
Disallow: /blog/
Allow: /blog/2024/
Allow: /blog/2025/

This blocks /blog/ in general but explicitly allows posts from 2024 and 2025.

Getting Re-Indexed After Fixing

After you update your robots.txt, Google will not immediately re-crawl the affected pages. Here is how to speed up the process.

Preventing Future Issues

A few practices to keep robots.txt problems from recurring:

• Version control your robots.txt. Keep it in your repository so changes are tracked and reviewable.
• Add robots.txt checks to your deployment pipeline. Validate the file automatically before every deploy.
• Monitor Google Search Console regularly. Check the Pages report at least monthly for new "Blocked by robots.txt" entries.
• Be cautious with wildcards. When adding wildcard rules, test them against a list of your important URLs to catch unintended matches.
• Document your rules. Add comments to your robots.txt explaining why each rule exists. Future you (or your teammates) will thank you.

# Block admin area
User-agent: *
Disallow: /admin/

# Block search results pages (thin content)
Disallow: /search

# Block API endpoints (not for public consumption)
Disallow: /api/

---

A blocked page cannot rank. Find the rule, fix it, and move on.