Robots.txt Generator
Build a valid robots.txt with common crawler toggles, custom disallow rules, and a sitemap pointer. Copy or download.
Pick the crawlers you care about — including AI bots like GPTBot, ClaudeBot, and PerplexityBot — set allow/disallow rules per agent, point to your sitemap, and copy a valid robots.txt for your site root. Built for founders and marketers shipping fast.
All crawlers (*)
Default rules for every bot
User-agent: * Disallow:
How it works
Save the output as robots.txt at your domain root (https://yoursite.com/robots.txt). Each block tells one user-agent what it may crawl. An empty Disallow: line means “allow everything”; Disallow: / blocks the whole site. Crawl-delay is honored by Bing and some bots but ignored by Google.
get the full growth toolkit
reddit-validated weekly distribution plans, ai-visibility tracking, and more — free to start.