robots.txt Testing for Content Publishers

You publish an investigative piece that took your team three months to report. It gets picked up across the web, drives significant traffic, and wins industry recognition. Six months later, you discover that the article -- along with thousands of others from your archive -- has been scraped and used to train a large language model. Your content is now being regurgitated in AI chatbot responses without attribution, without compensation, and without your permission.

You want to block AI training crawlers. But you also depend on Google and Bing for the majority of your traffic. You need search engines to find your content. The question is how to let the right crawlers in while keeping the wrong ones out.

The answer starts with your robots.txt file.

The publisher's dilemma

Publishers live in a tension that did not exist five years ago. Search engine crawlers and AI training crawlers both want access to your content, but for fundamentally different purposes.

You cannot use a single Disallow: / to solve this. That would block everything, including the search engines that drive your audience to your site. You need selective blocking -- allowing Googlebot and Bingbot while denying GPTBot, ClaudeBot, and other AI crawlers.

How robots.txt controls different crawler types

Every web crawler identifies itself with a User-agent string. Your robots.txt file can target rules to specific crawlers by name. This means you can write one set of rules for search engines and a completely different set for AI training bots.

# Allow search engines full access
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

# Default: allow everything else
User-agent: *
Allow: /

Sitemap: https://yourpublication.com/sitemap.xml

This configuration lets search engines crawl your entire site while blocking major AI training crawlers from accessing any of your content.

The AI crawlers you should know about

The landscape of AI crawlers is growing. Here are the major ones publishers should consider blocking or allowing based on their content strategy.

Testing that your rules actually work

Writing the rules is only half the job. You need to verify that they do what you intend. A typo in a User-agent name means the rule does nothing. A misplaced directive can block the wrong crawler.

Selective blocking strategies for publishers

Not every publisher wants to block all AI crawlers completely. Some want nuanced control. Here are common strategies.

Block all AI training, allow search. The most common approach. Block GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and CCBot. Allow Googlebot, Bingbot, and other search crawlers. This maximizes search visibility while denying AI training access.

Block AI training but allow AI search. Some publishers want to appear in AI-powered search results (like Perplexity or Google AI Overviews) but do not want their content used for model training. This requires understanding which crawlers serve which purpose -- Google-Extended is for AI training, while Googlebot handles both search and AI search features.

Block everything behind the paywall. If you have a metered or hard paywall, you might allow AI crawlers to access free content while blocking them from premium articles. Use path-based rules to protect specific directories.

Time-based approach. Some publishers block AI crawlers from recent content (the first 30 days) and allow access to archived material. While robots.txt itself does not support time-based rules, you can implement this by moving content between blocked and allowed paths as it ages.

Pricing

Robots.txt Tester is free. Test your publisher robots.txt against any crawler user agent, as often as you need.

---

Part of Boring Tools -- boring tools for boring jobs.