BYTETOOLS

Robots.txt Tester

Paste your robots.txt, a URL path and a user-agent to see if crawling is allowed or blocked, with the exact deciding rule highlighted. Free and private.

Allowed
/admin/public/report.html — deciding rule: Allow: /admin/public/ (line 4)

Matched group: User-agent: * (no named group matched)

LineRulePatternLengthMatches path
2Disallow/admin/7Yes
3Disallow/*.pdf$7No
4Allow/admin/public/14Yes

Sitemap: https://example.com/sitemap.xml

What is the Robots.txt Tester?

The Robots. txt Tester tells you whether a specific path is allowed or blocked for a given crawler, using the same Robots Exclusion Protocol rules that Googlebot follows: group selection by the most specific matching user-agent, longest-rule-wins precedence, * and $ wildcards, and Allow beating Disallow on ties.

  • Implements REP matching: longest rule wins, Allow beats Disallow on ties
  • Supports * wildcards and $ end-of-URL anchors
  • Selects the correct group for your user-agent, falling back to *
  • Highlights the exact rule that decided the verdict
  • Tests drafts and staging files — paste text, no fetching, no uploads
  • Lists Sitemap and Crawl-delay lines found in the file

How to use the Robots.txt Tester

  1. 1

    Paste the full contents of your robots.txt file into the text box.

  2. 2

    Enter the URL path you want to test, such as /blog/post-1 or a full URL.

  3. 3

    Enter the user-agent to test as, e.g. Googlebot or Bingbot, or * for generic crawlers.

  4. 4

    Read the Allowed or Blocked verdict and the highlighted deciding rule.

  5. 5

    Adjust your rules and re-test instantly until the behaviour is what you intended.

About the Robots.txt Tester

The Robots.txt Tester tells you whether a specific path is allowed or blocked for a given crawler, using the same Robots Exclusion Protocol rules that Googlebot follows: group selection by the most specific matching user-agent, longest-rule-wins precedence, * and $ wildcards, and Allow beating Disallow on ties.

Paste the contents of your robots.txt file into the box — this tool works entirely on pasted text and never fetches a live URL, so you can test drafts, staging copies or proposed changes before anything goes live. Copy the file straight from your editor or from yourdomain.com/robots.txt in another tab.

Everything runs 100% locally in your browser; nothing you paste is uploaded or stored. The verdict shows which group was selected for your user-agent and exactly which rule decided the outcome, so debugging an unexpected block takes seconds instead of guesswork.

Frequently asked questions

How does Google decide which robots.txt rule applies?

Google first picks the group whose User-agent line most specifically matches the crawler, then compares every Allow and Disallow rule in that group against the path. The rule with the longest matching path wins, and if an Allow and a Disallow tie in length, the Allow rule takes precedence.

What do the * and $ wildcards mean in robots.txt?

An asterisk matches any sequence of characters, so Disallow: /*.pdf blocks every URL containing .pdf after any prefix. A dollar sign anchors the rule to the end of the URL, so Disallow: /*.pdf$ only blocks URLs that actually end in .pdf and not /file.pdf?download=1.

Why can't this tool fetch my robots.txt from my domain?

This tester runs entirely in your browser with no server component, and browsers block cross-site fetches of other domains' files. Pasting the file contents is also more useful in practice: you can test unsaved drafts and proposed changes before deploying them.

Does Disallow in robots.txt remove a page from Google?

No. Disallow only stops crawling, not indexing — a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex robots meta tag, or protect the page with authentication.

Which group applies if my crawler matches several User-agent lines?

The most specific match wins. Googlebot-Image follows a User-agent: Googlebot-Image group over a User-agent: Googlebot group, and any named match beats the * group entirely — the rules are not combined across differently named groups.

Is an empty Disallow line the same as blocking everything?

The opposite. Disallow: with no value matches nothing, which means the group allows everything. Blocking the whole site is Disallow: / — a one-character difference that has taken down more than one site's search traffic, which is exactly why testing first matters.

Related tools