Reference
Syntax references, directive glossaries, example files, and bot user-agent lists.
Quick-reference material for robots.txt syntax, directives, and crawler user-agent strings. Useful when you know what you're looking for and need the exact format or value.
robots.txt Glossary: Every Directive and Term Explained
Definitions of every robots.txt directive and related term. User-agent, Disallow, Allow, Sitemap, Crawl-delay, and more.
Read morerobots.txt Examples for Every Situation
Copy-paste robots.txt examples: block all bots, allow all bots, block specific directories, WordPress, Shopify, e-commerce, and more.
Read morerobots.txt Syntax Reference
Complete robots.txt syntax reference. Every directive, pattern, and rule explained with examples. Bookmark this page.
Read moreComplete List of Search Engine and AI Bot User-Agents
A comprehensive reference of search engine crawler and AI bot user-agent strings. Covers Google, Bing, Yandex, Baidu, DuckDuckGo, and major AI crawlers with their robots.txt identifiers.
Read more