Robots.txt Tester - Check if a URL is Blocked

How the Robots.txt Tester Works

1. 01Paste your robots.txt content, or load one of the built-in templates as a starting point.
2. 02Enter the URL path you want to test and pick a user-agent: Googlebot, Bingbot, GPTBot, any bot, or a custom name.
3. 03The tester selects the matching user-agent group and applies the longest-match rule between Allow and Disallow, honoring * wildcards and $ anchors.
4. 04Read the ALLOWED or BLOCKED verdict with the deciding rule highlighted, plus every sitemap declared in the file.

Understanding How Crawlers Read Robots.txt

Robots.txt is deceptively simple and regularly misunderstood. A crawler first chooses exactly one group of rules: the group whose User-agent line most specifically matches its name, falling back to the asterisk group if none does. Rules from non-matching groups are ignored entirely, which surprises people who expect a Googlebot group and a star group to combine; they do not. Within the chosen group, the crawler compares the URL path against every Allow and Disallow pattern and obeys the longest matching pattern, with Allow winning exact ties. Wildcards make patterns flexible: an asterisk matches any sequence of characters, and a dollar sign anchors the pattern to the end of the URL, so Disallow: /*.pdf$ blocks PDF files everywhere. Three traps account for most real-world incidents. An empty Disallow line blocks nothing at all. Disallow: / blocks everything, and it has taken down entire sites when a staging file shipped to production. And robots.txt controls crawling, not indexing: a blocked URL can still appear in search results if other sites link to it, so use a noindex meta tag when you need a page out of results. Test every rule change here before deploying, exactly as the crawler will read it.

When to Use the Robots.txt Tester

Use it before every robots.txt deployment, when pages unexpectedly vanish from search results, and when Search Console reports blocked resources. It answers concrete questions fast: is /admin/ actually blocked, does the Allow exception for admin-ajax.php work, will GPTBot be kept out while Googlebot crawls freely? It is also the safe way to review a robots.txt copied from a template or a plugin, since a single wrong slash can hide your whole site. Because parsing happens locally, testing draft rules for staging and unreleased sections is completely private.

Common Use Cases

• —Verifying a new robots.txt before deployment so no critical section is accidentally blocked.
• —Debugging why a page disappeared from search by testing its exact path against the live rules.
• —Checking how AI crawlers like GPTBot are treated compared to Googlebot and Bingbot.
• —Confirming the sitemap declarations that search engines will discover in the file. Meta Tag Generator - SEO Tags in Seconds

Expert Tips

• ✻Test the most valuable paths on your site, not just the ones you meant to block; incidents come from unintended matches.
• ✻Remember rules match the path only: Disallow: /shop blocks /shop and /shopping alike unless you add a trailing slash.
• ✻Keep a permissive User-agent: * group even when blocking specific bots, so unknown crawlers get sane defaults.

Frequently Asked Questions

If a URL is blocked in robots.txt, is it removed from Google?→

Not necessarily. Robots.txt stops crawling, not indexing. A blocked URL can still appear in results, usually without a description, if other pages link to it. Use a noindex meta tag on a crawlable page to remove it properly.

Which rule wins when Allow and Disallow both match?→

The rule with the longest matching pattern wins. If an Allow and a Disallow pattern match with equal length, Allow wins. The tester highlights exactly which rule decided your verdict and on which line.

Do Googlebot rules and the * group combine?→

No. A crawler obeys only the single most specific matching group. If a Googlebot group exists, Googlebot ignores the * group completely, which is one of the most common robots.txt misunderstandings.