robots.txt Testing for Developers
Validate your robots.txt during development and deployment. Catch misconfigurations before they reach production.
You spin up a Next.js app, scaffold the pages, push through a few sprints, and deploy to production. Traffic looks fine at first. Two weeks later, the SEO team pings you: Google has not indexed a single page. You check the robots.txt and there it is -- the framework's default Disallow: / that you never updated. The file you never thought about just made your site invisible to every search engine on the internet.
This is one of the most common deployment mistakes developers make, and it is completely preventable.
Common developer robots.txt mistakes
Robots.txt is a file most developers create once and forget about. That is exactly why mistakes happen. Here are the patterns that cause the most damage.
Leaving staging rules in production
During development, you set Disallow: / to keep search engines away from your staging environment. That makes sense. The problem is when this file gets deployed to production without anyone changing it. Your live site now tells every crawler to go away.
Framework defaults
Many frameworks generate a robots.txt automatically. Some default to blocking everything. Others generate an empty file that provides no crawl guidance. If you do not check what your framework produces, you are deploying a file you have never read.
Wrong path patterns
A developer writes Disallow: /api intending to block the API endpoints. But that rule also blocks /api-documentation/, /api-status/, and any other path starting with /api. The missing trailing slash turns a targeted block into a broad one.
Dynamically generated robots.txt with bugs
Some apps generate robots.txt programmatically based on environment variables or configuration. A bug in the generation logic, a missing env var, or a bad conditional can produce a broken file that blocks things it should not.
Conflicting rules across environments
Your staging, preview, and production environments each need different robots.txt rules. Without a clear strategy for managing these, it is easy to mix them up during deployment.
Why robots.txt deserves a spot in your workflow
As a developer, you test your code, your APIs, and your database queries. You run linters and formatters. You write unit tests. But robots.txt -- a plain text file that controls whether your entire site appears in search results -- usually gets zero validation.
This is a small file with outsized impact. A single wrong line can:
- Block search engines from your entire application
- Prevent crawlers from accessing CSS and JavaScript they need to render your pages
- Accidentally expose staging URLs by not blocking them
- Break sitemap discovery by omitting or misconfiguring the Sitemap directive
| Without validation | With validation |
|---|---|
| Staging `Disallow: /` ships to production unnoticed | Tester flags the full-site block before deploy |
| Wildcard rules block unintended paths | Test specific URLs to confirm behavior |
| Framework defaults go unchecked | Validate generated output against expectations |
| Errors discovered weeks later via traffic drops | Errors caught in minutes during development |
| Debugging requires manual rule parsing | Instant feedback on allow/block status per URL |
Validate before you deploy
Paste your robots.txt and test it against specific URLs. Know exactly what crawlers will see before it goes live.
Integrating robots.txt testing into development
You do not need to overhaul your workflow. Robots.txt Tester fits into the process you already have.
Check your framework's default output
Before your first deployment, look at what robots.txt your framework generates. If you use Next.js, check the output of your robots.ts or robots.txt file in the public directory. For Nuxt, check nuxt.config. For static sites, check the build output. Paste the result into the tester and confirm it does what you expect.
Test during local development
When you update robots.txt rules, validate them immediately. Paste your file into Robots.txt Tester and test the URLs that matter: your homepage, key landing pages, API routes, and any paths you intend to block. This takes less than a minute.
Validate before deployment
Add robots.txt validation to your pre-deployment checklist. Before you push to production, confirm the file matches your production rules, not your staging or development rules. If your robots.txt is generated dynamically, test the output for the production environment specifically.
Verify after deployment
After deployment, fetch your live robots.txt and run it through the tester. This confirms that the file was deployed correctly and the server is serving it as expected. A quick post-deploy check catches issues from CDN caching, server configuration, or deployment scripts that modify files.
Quick sanity check for common scenarios
Here are the robots.txt checks every developer should run before shipping.
New project launch. Is Disallow: / anywhere in the production robots.txt? If so, it should not be there unless you intentionally want to block all crawlers.
Site migration. Do the rules account for the new URL structure? Old Disallow paths that no longer exist are harmless, but new paths that should be blocked might be missing.
Environment-specific rules. Is your staging environment blocked from crawlers? Is your production environment open? Are your preview URLs (Vercel, Netlify) blocked so Google does not index them?
Dynamic generation. Does the generated output match what you expect for each environment? Try logging the output and comparing it against your intended rules.
Watch out for CDN caching
If you use a CDN, your robots.txt might be cached even after you deploy a new version. After updating the file, purge the CDN cache for /robots.txt or wait for the TTL to expire. Test the live URL to confirm the new version is being served.
Test your robots.txt in 30 seconds
No signup, no installation. Paste your file, test your URLs, and confirm everything works.
Pricing
Robots.txt Tester is free. Use it as a quick sanity check during development, or as a regular part of your deployment process.
Free
$0
- Up to 3 items
- Email alerts
- Basic support
Pro
$9/month
- Unlimited items
- Email + Slack alerts
- Priority support
- API access
Share it with your team
Bookmark Robots.txt Tester and share it with your team. When anyone touches the robots.txt, they can validate their changes immediately instead of waiting for an SEO audit to catch mistakes weeks later.
Related Articles
Part of Boring Tools -- boring tools for boring jobs.
Test your robots.txt for free
Validate your robots.txt file instantly. Check directives, find crawling issues, and ensure search engines can access your site.