Robots.txt generator
Build a robots.txt file that allows or blocks crawlers and points to your sitemap.
🔒 Everything runs in your browser. Nothing you choose or type is uploaded. How to verify
How to use it
- Choose whether to allow or block everything.
- List paths to disallow and add your sitemap URL.
- Copy the text into a file named robots.txt at your site root.
Handy for
Setting up a new website or staging area.
Good to know
Block only what you must. Blocking all crawlers removes your site from search over time.
Privacy: where the work happens
Everything you enter or choose is processed by JavaScript in your browser and never sent anywhere. Only the page code, and any library this tool needs, is downloaded from cdnjs or jsDelivr.
Questions
Does robots.txt hide pages from Google?
Not reliably. It asks crawlers not to visit, but a blocked page can still be listed if others link to it. Use noindex for that.
Where does the file go?
At the root of your domain, for example example.com/robots.txt.