robots.txt tester
Check whether Google and AI crawlers like GPTBot, ClaudeBot and PerplexityBot may crawl a page, and which rule decides. Test your live file or paste one.
More free tools
- Sitemap checker Find your sitemap, check it's valid, and see whether the pages in it load and can be indexed. Free, no sign-up.
- Indexability checker Check a page for everything that keeps it out of Google: errors, redirects, robots.txt, noindex, canonical tags and text that only appears after JavaScript.
- Redirect checker See every step a link takes before it lands: 301s, 302s, chains and loops. We also check that the http, https, www and non-www versions of your site end at the same page.
- llms.txt generator Create an llms.txt for your site in seconds: we read your sitemap and pages and group your key pages for AI tools like ChatGPT and Claude. Copy it or download it.
- llms.txt validator Check your site's llms.txt: is it there, is it written in the right format, and do its links work.