Resources
Guides, tutorials, and tips for domain expiry monitoring.
Do You Need a robots.txt File?
Does your website need a robots.txt file? When it's essential, when it's optional, and what happens if you don't have one.
Read moreHow to Add Your Sitemap to robots.txt
Add your sitemap URL to robots.txt so search engines can find it. Syntax, multiple sitemaps, and common mistakes to avoid.
Read moreHow to Block AI Crawlers with robots.txt
Block AI training crawlers like GPTBot, ClaudeBot, and PerplexityBot using robots.txt. Complete list of AI user agents and copy-paste rules.
Read moreHow to Check Any Website's robots.txt File
How to find and check any website's robots.txt file. View directives, verify accessibility, and check for common configuration mistakes.
Read moreHow to Create a robots.txt File
Step-by-step guide to creating a robots.txt file. Learn the syntax, write directives for search engines, and deploy your file correctly.
Read moreHow to Edit robots.txt in Shopify
How to customize your Shopify store's robots.txt using the robots.txt.liquid template. Default rules, common customizations, and gotchas.
Read moreHow to Edit robots.txt in WordPress
Edit your WordPress robots.txt file using Yoast SEO, Rank Math, or direct file editing. Step-by-step instructions for each method.
Read moreHow to Fix "Blocked by robots.txt" Errors
Fix 'blocked by robots.txt' errors in Google Search Console. Diagnose Disallow rules, update your robots.txt, and get your pages indexed.
Read moreHow to Read and Understand robots.txt
Learn to read any robots.txt file. Understand User-agent, Disallow, Allow, Sitemap, and Crawl-delay directives with real-world examples.
Read moreHow to Test and Validate Your robots.txt
Four ways to test your robots.txt file: online validators, Google Search Console, command-line tools, and dedicated testing services.
Read moreHow robots.txt Affects Your SEO
How robots.txt impacts search engine optimization. Crawl budget, indexing, and the SEO mistakes that robots.txt can cause.
Read morerobots.txt Best Practices
Robots.txt best practices for SEO and crawl management. What to block, what to allow, and the mistakes that hurt your site.
Read moreCrawl-Delay in robots.txt Explained
What the Crawl-delay directive does, which search engines support it, and when you should (and shouldn't) use it.
Read morerobots.txt Glossary: Every Directive and Term Explained
Definitions of every robots.txt directive and related term. User-agent, Disallow, Allow, Sitemap, Crawl-delay, and more.
Read morerobots.txt Disallow Directive Explained
How the Disallow directive works in robots.txt. Syntax, examples, path matching, and common mistakes that accidentally block your entire site.
Read morerobots.txt Examples for Every Situation
Copy-paste robots.txt examples: block all bots, allow all bots, block specific directories, WordPress, Shopify, e-commerce, and more.
Read moreWhy You Should Monitor Your robots.txt for Changes
Accidental robots.txt changes can deindex your entire site. Why monitoring your robots.txt matters and how to set it up.
Read moreNoindex in robots.txt: Why It Doesn't Work
Google no longer supports the noindex directive in robots.txt. What happened, what to use instead, and how to properly deindex pages.
Read morerobots.txt SEO Audit Checklist
A checklist for auditing your robots.txt file. Check for SEO issues, crawl problems, and misconfigurations in under 10 minutes.
Read morerobots.txt Syntax Reference
Complete robots.txt syntax reference. Every directive, pattern, and rule explained with examples. Bookmark this page.
Read morerobots.txt User-Agent: How to Target Specific Crawlers
How to use the User-agent directive in robots.txt to create rules for specific search engines, bots, and crawlers.
Read morerobots.txt vs Meta Robots Tags: Which to Use
The difference between robots.txt and meta robots tags (noindex, nofollow). When to use each, and why using the wrong one can hurt your SEO.
Read moreWildcards in robots.txt: Using * and $ Patterns
How to use wildcard patterns in robots.txt. The * and $ characters, path matching, and practical examples for complex blocking rules.
Read moreWhat Does robots.txt Actually Do?
What robots.txt does and doesn't do. How crawlers use it, why it's advisory not enforceable, and the limits of robots.txt.
Read moreWhat Is robots.txt?
What robots.txt is, how it works, and why every website should have one. The complete introduction to the robots exclusion protocol.
Read more