How to Edit robots.txt in WordPress

WordPress handles robots.txt differently from most platforms. By default, it generates a virtual robots.txt file dynamically -- there is no physical file on disk. This catches many developers off guard when they try to edit it directly.

This guide covers three methods to edit your WordPress robots.txt: through Yoast SEO, through Rank Math, and by creating a physical file. Each approach has trade-offs.

How WordPress Handles robots.txt

WordPress generates a virtual robots.txt file automatically. When a crawler requests https://yoursite.com/robots.txt, WordPress intercepts the request and outputs a default set of rules.

The default WordPress virtual robots.txt looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/wp-sitemap.xml

This blocks crawlers from the WordPress admin area (except the AJAX endpoint, which some themes and plugins need) and points to the built-in WordPress sitemap.

Key things to understand:

• There is no physical robots.txt file in your WordPress installation by default. The file is generated on-the-fly.
• If you create a physical file, WordPress will serve that instead of the virtual one.
• Plugins can modify the virtual file by hooking into the robots_txt filter.
• If your site is set to "Discourage search engines" (Settings > Reading), WordPress adds Disallow: / to the virtual file, blocking all crawlers from everything.

Method 1: Editing with Yoast SEO

Yoast SEO provides a built-in robots.txt editor. This is the most popular method and does not require FTP access.

If you do not see the "File editor" option in Yoast, your hosting environment may have file editing disabled. Check if DISALLOW_FILE_EDIT is set to true in your wp-config.php.

Method 2: Editing with Rank Math

Rank Math also includes a robots.txt editor, and it works similarly to Yoast's.

Rank Math also provides a robots.txt validation feature that checks for basic syntax errors before saving.

Method 3: Direct File Editing

If you do not use Yoast or Rank Math, or if you prefer full control, you can create and edit the robots.txt file directly.

Via FTP or SFTP:

1. Connect to your server using an FTP client (FileZilla, Cyberduck, etc.)
2. Navigate to your WordPress root directory (the same directory that contains wp-config.php)
3. Create or edit the robots.txt file
4. Upload the file

Via your hosting file manager:

Most hosting control panels (cPanel, Plesk, etc.) include a file manager. Navigate to your WordPress root directory and create or edit the robots.txt file.

Via SSH:

# Connect to your server
ssh user@yourserver.com

# Navigate to WordPress root
cd /var/www/html

# Create or edit robots.txt
nano robots.txt

When you create a physical file, it completely replaces the virtual robots.txt. WordPress will not add its default rules or sitemap reference automatically. You are responsible for the entire file content.

WordPress-Specific robots.txt Rules

Here is a well-configured robots.txt for a typical WordPress site:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-json/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /readme.html
Disallow: /license.txt
Disallow: /?s=
Disallow: /search/
Disallow: /author/
Disallow: /*?replytocom=
Disallow: /tag/*/page/
Disallow: /category/*/page/

# Allow CSS, JS, and images for rendering
Allow: /wp-content/uploads/
Allow: /wp-content/themes/

Sitemap: https://yoursite.com/wp-sitemap.xml
Sitemap: https://yoursite.com/sitemap_index.xml

Let me break down the reasoning:

• /wp-admin/ and /wp-includes/ -- Backend files that should not be indexed
• /wp-content/plugins/ -- Plugin files that should not appear in search results
• /wp-json/ -- REST API endpoints that produce raw JSON, not user-facing pages
• /?s= and /search/ -- Internal search results, which are thin content
• /author/ -- Author archive pages, often low-value duplicate content on single-author sites
• /*?replytocom= -- Comment reply URLs that create duplicate page versions
• /tag/*/page/ and /category/*/page/ -- Paginated tag and category archives

What You Should Never Block in WordPress

Handling WooCommerce

If you run WooCommerce, add these rules to prevent crawling of cart, checkout, and account pages:

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=*
Disallow: /*?orderby=*
Disallow: /*?filter_*

These pages are session-specific and should not be indexed. The filter and orderby parameters create thousands of duplicate URLs that waste crawl budget.

Virtual vs. Physical File: Which to Use

For most WordPress sites, using a plugin editor is the best balance of convenience and control. If you manage your site through version control and deployment pipelines, a physical file that you maintain in your repository is the cleanest approach.

---

WordPress makes robots.txt easy to edit, but easy to break. Test after every change.