Robots.txt Analyzer
Paste your robots.txt to see exactly how crawlers will read it — user-agent groups, allow/disallow rules, sitemaps — and get warned about rules that block your whole site.
🔒 Runs entirely in your browser · nothing is uploaded or stored
⚠️ The classic mistake
Disallow: / under User-agent: * blocks your entire site from search engines. This tool flags it loudly.
📘 Quick rules
- robots.txt controls crawling, not indexing
- Use
noindexmeta to keep pages out of results - Always list your
Sitemap: - Blocked pages can still appear if linked
Frequently Asked Questions
Does Disallow remove a page from Google?
No. Disallow only stops crawling. A disallowed URL can still be indexed (shown without a snippet) if other pages link to it. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header.
Where should robots.txt live?
At the root: https://yourdomain.com/robots.txt. It only applies to its own host and protocol.
Related tools