Robots.txt Generator
Control how search engine bots crawl and index your website.
General Settings
Most modern sites don't need this.
Directives (Rules)
ROBOTS.TXT PREVIEW
About Robots.txt Generator Tool
Create a robots.txt file to instruct search engine bots on which parts of your site they are allowed to crawl.
How to Use Robots.txt Generator
- Use the "Quick Presets" for WordPress or Shopify if applicable.
- Add specific "Allow" or "Disallow" rules for paths.
- Add your Sitemap URL at the bottom.
- Download and upload to your root folder (e.g., example.com/robots.txt).
Frequently Asked Questions
What happens if I don't have one?
Bots will assume they are allowed to crawl everything on your site.
Can I block Google completely?
Yes, using "User-agent: * Disallow: /". Only use this for dev sites!
What is Crawl-delay?
It tells bots to wait X seconds between requests to save server load. Google ignores this, but Bing uses it.