Free Robots.txt Generator — Beginner Friendly, Pro Powerful

WordPress sites have three good options depending on which SEO plugin you run. Pick one method — do not combine them.

Method 1 — Yoast SEO plugin

1. WordPress Admin → Yoast SEO → Tools
2. Click File editor
3. If no robots.txt exists, click Create robots.txt file
4. Paste the generated content from the preview panel above
5. Click Save changes to robots.txt and verify at yourdomain.com/robots.txt

Method 2 — Rank Math plugin

1. WordPress Admin → Rank Math → General Settings
2. Click Edit robots.txt in the sidebar
3. Paste the generated content
4. Click Save Changes

Method 3 — Manual via FTP / cPanel

1. Click Download robots.txt in the preview panel above
2. Connect via FTP or cPanel File Manager
3. Upload to the WordPress root folder (same level as wp-config.php)
4. Verify at yourdomain.com/robots.txt

Shopify auto-generates a sensible robots.txt for every store. To customise it, you must create a Liquid template that extends the default.

1. Shopify Admin → Online Store → Themes
2. Click the three-dot menu on your live theme → Edit code
3. Under Templates, click Add a new template
4. Choose robots.txt from the dropdown and click Create template
5. Replace the default content with the Liquid snippet below (it loops Shopify's defaults then adds your custom rules)
6. Click Save and verify at yourdomain.com/robots.txt

# Generated by SEO First Web — seofirstweb.co.uk/tools/robots-txt-generator
{%- for group in robots.default_groups -%}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /your-custom-path/' }}
  {%- endif -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{%- endfor -%}

Wix exposes a basic robots.txt editor on Business / Premium plans.

1. Site Dashboard → Marketing & SEO → SEO Tools
2. Click Robots.txt Editor
3. Paste the generated content
4. Click Save

Squarespace auto-generates robots.txt and intentionally limits customisation. You have two real options:

Option A — built-in crawler settings

1. Settings → Crawlers
2. Toggle each crawler on or off (basic allow/block per major bot)
3. Save

Option B — Developer Mode (full custom robots.txt)

To deploy a fully custom robots.txt you must switch your site to Developer Mode and add a robots.txt file at the site root in the Git repository. This is a one-way change for the underlying site files — back up first.

Webflow gives you a clean robots.txt editor on every paid plan.

1. Project Settings → SEO tab
2. Scroll to Robots.txt
3. Paste the generated content
4. Click Save Changes
5. Click Publish to push the change to your live domain — robots.txt updates do NOT go live until you publish

Ghost serves a default robots.txt automatically. To replace it:

Self-hosted Ghost

1. SFTP into your Ghost host
2. Navigate to /content/static/ (create it if it does not exist)
3. Upload your generated robots.txt here
4. Restart Ghost to clear any cached default

Ghost(Pro) hosted

You cannot upload arbitrary files. The supported workaround is a routes.yaml override that maps /robots.txt to a route on your theme. See the official Ghost routing docs for the syntax.

The classic deployment — works on every shared host, every static host, and every CDN.

1. Click Download robots.txt in the preview panel above
2. Connect to your host via FTP, SFTP, or the cPanel File Manager
3. Upload robots.txt to the web root (the same folder that contains index.html)
4. Verify at yourdomain.com/robots.txt

For sites running on Apache (the most common Linux web server).

1. Save the generated file as robots.txt
2. Place in your document root — typically /var/www/html/
3. No .htaccess changes are needed; Apache serves robots.txt automatically
4. Verify: curl https://yourdomain.com/robots.txt

For sites running on Nginx — common for VPS and high-traffic deployments.

1. Save the generated file as robots.txt
2. Place in your document root — typically /usr/share/nginx/html/ or /var/www/
3. Optional but recommended — add an explicit location block so logs do not pollute and the correct Content-Type is set
4. Reload Nginx: sudo nginx -s reload

location = /robots.txt {
  add_header Content-Type text/plain;
  log_not_found off;
  access_log off;
}

Static method (recommended for most projects)

1. Save the generated file as robots.txt
2. Place it at /public/robots.txt in your Next.js project
3. Commit and push — Vercel auto-deploys
4. Verify at yourdomain.com/robots.txt

Dynamic method — Next.js App Router

Use this when your robots.txt needs to vary between staging and production environments. Create app/robots.ts:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Static method

1. Save the generated file as robots.txt
2. Place it at /public/robots.txt in your Astro project
3. Commit and push — your Astro host (Vercel, Netlify, Cloudflare Pages) auto-deploys
4. Verify at yourdomain.com/robots.txt

Dynamic method — Astro endpoint

Create an endpoint at src/pages/robots.txt.ts for environment-dependent output:

import type { APIRoute } from 'astro';

const robotsTxt = `
User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: ${import.meta.env.SITE}/sitemap-index.xml
`.trim();

export const GET: APIRoute = () =>
  new Response(robotsTxt, {
    headers: { 'Content-Type': 'text/plain' },
  });

For sites proxied through Cloudflare (orange-cloud DNS).

Default — origin serves it

Your origin server's robots.txt is served as normal through Cloudflare's proxy. No additional configuration required.

Override at the edge — Cloudflare Workers

1. Cloudflare dashboard → Workers & Pages → Create application
2. Add a Worker route for yourdomain.com/robots.txt
3. Return the generated content as a text/plain response
4. This lets you serve a different robots.txt to specific country regions or bot user-agents