Robots.txt Generator

A misconfigured robots.txt file can silently kill your search traffic. No error messages, no warnings — just pages that never get crawled, never get indexed, and never rank. Follow these best practices to ensure your robots.txt works for you, not against you.

Start Permissive, Then Restrict

The safest default is to allow everything and then block specific paths you don't want crawled. Start with User-agent: * and Allow: /, then add targeted Disallow rules for admin areas, staging pages, internal search results, and other low-value content. This approach prevents the common mistake of accidentally blocking important pages.

Always Include Your Sitemap

Adding a Sitemap directive is the single easiest SEO win you can get from robots.txt. It takes one line — Sitemap: https://yourdomain.com/sitemap.xml — and it ensures that every crawler, from Googlebot to the smallest niche search engine, knows exactly where to find your complete page listing. If you have multiple sitemaps (blog, products, pages), list all of them.

Block Low-Value URL Patterns

Internal search result pages, faceted navigation URLs, print versions, paginated archives, and URL parameter variations all waste crawl budget. Use wildcard patterns to block them efficiently. For example, Disallow: /search* blocks all internal search pages, and Disallow: /*?sort= blocks sort-parameter URLs. This focuses crawler attention on your canonical, high-value pages.

Handle AI Crawlers Deliberately

The rise of AI crawlers has added a new dimension to robots.txt management. Bots like GPTBot (OpenAI), Google-Extended (Google AI training), CCBot (Common Crawl), anthropic-ai (Anthropic), and Applebot-Extended (Apple Intelligence) scrape web content for training large language models. Decide your policy: block all AI crawlers, allow specific ones, or allow everything. Whatever you choose, make it a deliberate decision rather than a passive default.

Don't Block CSS and JavaScript

This was common advice in the early 2000s but is actively harmful today. Google needs to render your pages to understand them fully. If you block CSS and JavaScript files in robots.txt, Googlebot can't render your page, which means it can't evaluate your content layout, user experience, or mobile friendliness. Always allow crawling of CSS and JS resources.

Use Crawl-delay Wisely

The Crawl-delay directive throttles how frequently a bot makes requests. It's useful for reducing server load from aggressive crawlers — but there's a catch. Googlebot ignores Crawl-delay entirely. To control Google's crawl rate, use the crawl rate settings in Google Search Console. Bingbot, Yandex, and most other crawlers do respect Crawl-delay. A value of 1-10 seconds is typical; anything higher risks slowing crawl discovery significantly.

Test Before and After Deploying

Before uploading a new robots.txt, test it using Google Search Console's robots.txt Tester. Enter URLs you expect to be blocked and URLs you expect to be allowed, and verify the tool shows the correct result for each. After deploying, re-test to confirm the live file matches what you intended. A typo in a single line can inadvertently block your entire site.

Remember: Robots.txt Is Not Security

This cannot be emphasized enough. The Robots Exclusion Protocol is voluntary. Well-behaved crawlers follow it; malicious bots, scrapers, and security scanners ignore it completely. Your robots.txt file is publicly accessible — anyone can read it and see which paths you're trying to hide. Never rely on robots.txt to protect sensitive information. Use proper authentication, server-side access controls, and firewall rules instead.

Keep It Simple and Maintainable

A robots.txt file with 200 lines of rules is a maintenance nightmare. Group your rules logically: one block for all-bot rules, one for AI crawlers, one for specific search engine exceptions. Add comments (lines starting with #) to explain why each rule exists. When you revisit the file in six months, those comments will save you from accidentally breaking something.