robots.txt Generator

Generate a robots.txt file for your website. Control which pages search engines can crawl and index.

User-agent

Use * for all bots, or specify e.g. Googlebot

Common Rules

Allow all (/)Disallow /adminDisallow /privateDisallow /apiDisallow /tmp

Custom Disallow Paths

One path per line

Sitemap URL

Crawl-delay (seconds)

Optional. Not all bots respect this directive.

Generated robots.txt

User-agent: *
Allow: /

Tip: Place this file at the root of your domain as /robots.txt. It must be accessible at https://yourdomain.com/robots.txt.

Wat is robots.txt?

robots.txt is een bestand in de root van je website dat zoekmachine-crawlers vertelt welke pagina's ze mogen indexeren en welke niet. Het is essentieel voor SEO — je kunt admin-pagina's, dubbele content en gevoelige gebieden blokkeren.