Technical SEO / robots-txt
robots.txt tester
See what you allow crawlers to fetch
Googlebot reads robots.txt before it reads your content. A stray Disallow: / , a missing Sitemap line, or rules that only apply to * while Googlebot is open can waste crawl budget or hide the site entirely.
What this checks
- Whether /robots.txt returns HTTP 200
- User-agent groups, Allow, and Disallow paths
- Sitemap: directives
- The raw file, with real line breaks
Why it matters
This file is the gate. If it is missing, empty, or wrong, you are guessing how bots treat /api/, /admin/, and parameter URLs. Fetch it, parse Allow/Disallow by user-agent, and confirm sitemap locations.
Use this page directly, or call check_robots_txt through WebMCP with the same URL.