Updated SEO

Robots.txt Tester

Paste a website URL, pick a bot, and test whether a page path is allowed or blocked. See the exact group and rule that matched.

Allowed vs Blocked Matched rule Wildcards Batch testing

Robots Rule Checker

Enter a site, choose a user-agent, and test one or many paths. The preview updates after robots.txt is fetched.

Paste a website URL and click “Fetch robots.txt”. The tool will auto-test after robots.txt loads.

Detected Directives

Sitemaps

—

Crawl-delay

—

Groups

—

Crawl-delay and wildcard support varies by crawler. Always verify behavior for the bot you care about.
Batch testing uses the robots.txt currently loaded from the website URL in the Single Test tab.
Paste a list of paths (or full URLs), then run the batch test.

Robots.txt Source

No robots.txt loaded yet.

Robots.txt Preview

—

How This Tester Chooses a Rule

  1. Fetch /robots.txt from the website URL you entered.
  2. Parse it into user-agent groups (each group has one or more user-agent names and rules).
  3. Pick the best-matching group for your user-agent (most specific match beats *).
  4. Find all rules that match your path, then choose the most specific rule (longest match).
  5. If there’s a tie, Allow wins over Disallow.
Crawler behavior varies. This tool follows common interpretation with wildcard support when enabled.

What If My Result Looks Wrong?

  • Check whether you tested the correct user-agent (e.g., Googlebot vs Googlebot-Image).
  • Confirm whether your rule uses * and $ and whether wildcard mode is enabled.
  • Make sure you tested the right path format (leading /).
  • Remember: robots.txt controls crawling, not indexing. “Allowed” does not mean “indexed.”

Related Calculators