Robots.txt Generator
Generate a robots.txt file to control how search engine crawlers access your website.
Robots.txt Settings
Generated robots.txt
What Is a robots.txt File?
A robots.txt file provides instructions to search engine crawlers about which areas of a website they may or may not access.
How to Use the Robots.txt Generator
Enter your desired user-agent, allowed paths, blocked paths, and sitemap URL. The generator creates a robots.txt file that you can copy and use on your website.
Important
Robots.txt rules control crawler access but are not a reliable method for preventing a page from appearing in search results. Use appropriate noindex directives when you need to control search engine indexing.