Web Tool Bench Robots.txt Tester
SeoCheck if Googlebot, Bingbot or GPTBot can crawl a path, and see the rule that decided it
About the tool
Enter a domain and a path, pick a crawler (Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot or any), and the Robots.txt Tester fetches the live robots.txt and tells you whether that path is allowed, showing the exact rule that matched. That matters because the longest matching rule wins regardless of order, and a crawler obeys only the most specific User-agent group, so adding a single Googlebot rule can silently discard every wildcard rule for Googlebot. The page also explains why Disallow stops crawling but not indexing. Free, no sign-up, nothing stored.
💬 Discussion (0)
No discussions yet. Be the first to start the conversation!