← All tools
robots.txt Generator
Generate a robots.txt file for your website. Control which pages search engines can crawl and index.
Use * for all bots, or specify e.g. Googlebot
One path per line
Optional. Not all bots respect this directive.
User-agent: * Allow: /
Tip: Place this file at the root of your domain as
/robots.txt. It must be accessible at https://yourdomain.com/robots.txt.What is robots.txt?
robots.txt is a file at the root of your website that tells search engine crawlers which pages to index and which to ignore. It's essential for SEO — you can block admin pages, duplicate content, and sensitive areas from being crawled.