robots.txt for Wix, Squarespace, and Webflow

Website builders handle robots.txt differently from traditional hosting. With a standard web server, you upload a text file to your root directory and move on. With Wix, Squarespace, and Webflow, the platform controls the file, and your ability to edit it ranges from nonexistent to nearly full control depending on which platform you are using.

This guide covers how each platform generates its robots.txt, what the defaults include, how to customize rules where possible, and the workarounds for platforms that lock you out. For background on how robots.txt works in general, see the robots.txt Guide.

Wix robots.txt

Wix generates your robots.txt automatically. You can view yours at https://yourdomain.com/robots.txt.

What the Default Includes

A typical Wix robots.txt looks something like this:

User-agent: *
Disallow: /_api/
Disallow: /sitemap-video.xml
Disallow: /_partials
Sitemap: https://yourdomain.com/sitemap.xml

Wix blocks its internal API routes and partial-render paths. It also includes your sitemap reference automatically. The defaults are reasonable for most sites. Wix has improved its robots.txt over the years, and the current version is much cleaner than what the platform produced in its early days.

Customization Options

Wix added the ability to edit robots.txt through the SEO settings panel. Here is how to access it:

1. Go to your Wix dashboard.
2. Navigate to Settings > SEO > SEO Tools.
3. Find the robots.txt editor.
4. Add or modify rules.

You can add custom User-agent, Disallow, and Allow directives. Wix appends your custom rules to its auto-generated defaults. You cannot remove Wix's built-in rules, but you can add rules on top of them.

Common Wix Customizations

Block AI crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Block specific pages:

User-agent: *
Disallow: /thank-you
Disallow: /internal-landing

Wix Gotchas

Wix periodically updates its auto-generated rules. If Wix changes its default robots.txt structure, your custom rules still apply, but the base rules around them may shift. Check your live robots.txt after any platform update.

Wix also manages redirects and URL structures in ways that can interact with your robots.txt rules. If you change a page slug, your Disallow rule for the old slug stops doing anything. There is no warning when this happens.

If you are using Wix's multilingual feature, each language version may have its own URL paths. Make sure your Disallow rules account for language prefixes like /es/ or /fr/ if you need to block translated versions of specific pages.

Squarespace robots.txt

Squarespace takes a more restrictive approach. The platform generates your robots.txt automatically, and direct editing is not available through the standard interface.

What the Default Includes

A typical Squarespace robots.txt:

User-agent: *
Disallow: /config
Disallow: /search
Disallow: /account
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xml

Squarespace blocks its configuration pages, search results, account pages, and API endpoints. It also includes the sitemap reference. These defaults are sensible and match what most SEO practitioners would recommend.

Can You Edit It?

No. Squarespace does not provide a robots.txt editor. There is no admin panel option, no file upload, and no code injection method that affects the robots.txt output.

This has been a longstanding limitation. Squarespace's position is that the default rules cover what most sites need, and giving users direct access to robots.txt creates more problems than it solves. That stance is frustrating for anyone who needs custom rules.

Workarounds

Since you cannot edit robots.txt on Squarespace, you need alternative approaches.

Use meta robots tags. Squarespace lets you add custom code to individual page headers through Settings > Advanced > Code Injection (per-page). Add a meta robots tag to control indexing:

<meta name="robots" content="noindex, nofollow">

This does not prevent crawling (the bot still visits the page), but it prevents indexing. For most purposes, this achieves the same result as a Disallow rule. See robots.txt vs meta robots for the distinction between blocking crawling and blocking indexing.

Password-protect pages. Squarespace lets you password-protect individual pages. Password-protected pages return a login wall to crawlers, which effectively prevents indexing.

Use the "Hide from search engines" toggle. Squarespace offers a per-page option to discourage search engines from indexing a page. This adds a noindex meta tag to the page. It is the closest thing to a Disallow rule that Squarespace offers.

Block AI crawlers with response headers. If you need to block specific bots like GPTBot, and your Squarespace site uses a CDN or reverse proxy in front of it (such as Cloudflare), you can add X-Robots-Tag headers at the CDN level. This is an advanced configuration and only works if your DNS setup supports it.

Squarespace Gotchas

The biggest issue is that you cannot block AI crawlers with robots.txt directly on Squarespace. If blocking AI training crawlers is important to you, Squarespace currently does not offer a native solution. You need an external proxy or CDN that lets you add custom response headers.

Squarespace also does not support Crawl-delay. If aggressive bots are hammering your site, you have no robots.txt mechanism to slow them down.

Webflow robots.txt

Webflow gives you the most control of the three platforms. You can write a fully custom robots.txt file.

What the Default Includes

By default, Webflow generates a basic robots.txt:

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

This is permissive by design. Everything is allowed unless you specify otherwise.

Customization Options

Webflow lets you edit your robots.txt directly:

1. Open your project in the Webflow Designer.
2. Go to Project Settings > SEO.
3. Find the robots.txt section.
4. Write your custom robots.txt content.
5. Publish your site.

You have full control over the file contents. You can write any valid robots.txt syntax, including multiple User-agent blocks, wildcard patterns, Allow and Disallow rules, Sitemap references, and Crawl-delay directives.

Example Webflow robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /staging/
Disallow: /thank-you

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Webflow Gotchas

When you customize the robots.txt in Webflow, your custom content replaces the default entirely. If you forget to include the Sitemap directive, it will not be there. If you forget the User-agent: * block, no general rules will apply. Always include the basics.

Webflow staging subdomains (.webflow.io) have their own robots.txt that blocks all crawlers by default. This is good -- you do not want staging content indexed. But if you are testing your robots.txt on the staging subdomain, you will not see your custom rules reflected there.

Changes to robots.txt in Webflow only take effect after you publish. If you save changes in the Designer but do not publish, the live robots.txt remains unchanged.

Platform Comparison

Here is a quick summary of how the three platforms compare.

Wix offers partial customization. You can add rules through the SEO settings panel, but you cannot remove Wix's default rules. Good enough for most use cases.

Squarespace offers no direct robots.txt editing. You are limited to meta robots tags, page-level settings, and external workarounds. The most restrictive of the three platforms.

Webflow offers full customization. You can write any valid robots.txt content. The most flexible option, but you are responsible for including everything your site needs.

Common Issues Across All Three Platforms

Forgetting to Check the Live File

After making changes on any platform, always visit https://yourdomain.com/robots.txt in your browser to verify what is actually being served. The admin panel may suggest one thing while the live file says another. See how to test your robots.txt for a thorough testing process.

Custom Domain vs Platform Subdomain

All three platforms assign a default subdomain (e.g., yoursite.wixsite.com, yoursite.squarespace.com, yoursite.webflow.io). The robots.txt for the subdomain and your custom domain may differ. Always check the robots.txt on your actual production domain, not the platform subdomain.

Assuming Disallow Means Noindex

A Disallow rule in robots.txt prevents crawling. It does not remove a page from search results if Google already knows about it through other links. If a page is already indexed and you block it with Disallow, Google may keep the URL in search results (with a "No information is available for this page" note) rather than removing it. For proper deindexing, use a noindex meta tag or X-Robots-Tag header instead. The robots.txt best practices article covers this distinction in detail.

Platform Migrations

If you migrate from one platform to another, your robots.txt rules do not carry over. A site moving from Webflow (where you had custom rules) to Squarespace (where you cannot customize) will lose those rules. Plan for this during migration. Document your current robots.txt, then figure out how to replicate those rules on the new platform using whatever methods are available.

When to Consider a Different Approach

If your platform does not support the robots.txt customization you need, consider these alternatives:

Meta robots tags handle per-page indexing control on all three platforms. They do not block crawling, but they prevent indexing, which is often what you actually want.

CDN-level controls (like Cloudflare Workers or Netlify edge functions) can intercept requests to /robots.txt and serve a custom file, regardless of what the platform generates. This is an advanced setup but gives you full control on any platform.

X-Robots-Tag headers can be added at the CDN or proxy level to control indexing for specific paths or file types without touching robots.txt at all. See our guide to creating a robots.txt for more on alternative approaches.

References

1. Wix Help Center - robots.txt
2. Squarespace Help - SEO and search
3. Webflow University - robots.txt
4. Google Search Central - robots.txt