Robots.txt Generator
Create a robots.txt file to control how search engines and bots access your site's pages
Block specific AI bots (optional)
What is the Fastols Robots.txt Generator?
This tool generates a Robots.txt file that controls how search engines crawl your site's pages, by setting a default allow or disallow rule, along with custom paths and specific bots that can be blocked.
This tool helps site owners control search engine behavior toward certain parts of the site, such as admin pages or internal files.
Built Without Any Server Call
Your robots.txt rules are assembled directly in the browser from the options you choose.
Live File Preview
See the exact robots.txt output update as you toggle each rule.
Copy-Paste Ready, Always Free
Download or copy the finished file with no sign-up needed.
How to Create a Robots.txt File in Simple Steps
- 1Choose the default rule: allow all bots, disallow them entirely, or set custom paths.
- 2Enter custom allow and disallow paths, plus a crawl-delay if you want to set one.
- 3Select any specific bots you want to block from the available bot list.
- 4Add your Sitemap URL if you have one, then copy the file's content or download it directly as robots.txt.
Frequently Asked Questions about the Robots.txt Generator
What default rule options are available for the Robots.txt file?
You can choose to allow all bots to crawl, disallow them entirely, or set custom allow and disallow paths.
How do I add custom allow and disallow paths?
Enter the paths in the allow and disallow fields, one path per line, and the tool automatically adds a leading slash to any path that doesn't start with one.
Can I block specific crawler bots by name?
Yes, the tool lets you select specific bots from a ready-made list to block separately from the site's general rule.
Can I add my Sitemap URL to the resulting file?
Yes, there is a dedicated field for entering your Sitemap URL, which is automatically added as a separate line at the end of the Robots.txt file.
How can I get a Robots.txt file ready to upload to my site?
You can copy the file's content directly from the result box, or click the download button to get a robots.txt file ready to upload to your site's root.