Robots.txt Tester

Fetch and analyze any website's robots.txt file to understand crawl directives.

Enter a URL above to get started

Pro tips
  • • "Disallow: /" blocks all crawlers — check this isn't accidentally set.
  • • Always declare your sitemap URL in robots.txt.
  • • Test individual URL paths in Google Search Console.