Robots.txt Tester
Fetch and analyze any website's robots.txt file to understand crawl directives.
Enter a URL above to get started
Pro tips
- • "Disallow: /" blocks all crawlers — check this isn't accidentally set.
- • Always declare your sitemap URL in robots.txt.
- • Test individual URL paths in Google Search Console.