Free tools · SEO crawl
robots.txt & sitemap checker
See whether crawlers can find your rules and XML sitemap — Sitemap: lines, Disallow traps, URL counts and plain-English issues for launches and migrations.
Why crawl files matter
robots.txt and sitemaps after every launch
A missing Sitemap: line, an accidental Disallow: / , or a 404 on /sitemap.xml can slow indexing for weeks. Search engines still discover some URLs without a sitemap, but crawl budget and launch checklists get messy when the public crawl files are wrong or missing.
This free robots.txt and XML sitemap checker pulls the same public files bots request — useful after WordPress installs, host moves, redesigns and staging-to-live cutovers. Pair it with the SEO meta checker for titles and Open Graph, and the HTTP status tool for redirects and soft 404s.
Jamie Freeman is a UK web designer and developer specialising in WordPress, WooCommerce, custom PHP, technical SEO and hosting. Free tools help you spot crawl traps; paid work is for fixing robots, sitemaps and site structure properly.
FAQ
robots.txt & sitemap FAQs
What does this robots.txt and sitemap checker do?
It fetches /robots.txt for a public domain, parses User-agent, Allow, Disallow, Crawl-delay and Sitemap lines, then tries common sitemap locations and any URLs declared in robots. You get a crawl-health grade, issue list, raw robots body and a sample of sitemap URLs.
Why is a Sitemap: line in robots.txt important?
The Sitemap directive tells crawlers where your XML sitemap lives. Without it (and without Search Console submission), bots may discover pages more slowly after a redesign or new host. Declaring the sitemap is a cheap win on every launch checklist.
What is an accidental Disallow: / ?
Disallow: / under User-agent: * tells all well-behaved bots not to crawl the site. Staging copies sometimes ship with that rule still on. This checker surfaces heavy disallow rules and missing robots files so you catch blocks before Google does.
Does a green sitemap mean I am fully indexed?
No. A valid sitemap and robots file help discovery; indexing still depends on content quality, links, status codes and Search Console. Pair this tool with the SEO meta checker and HTTP status tool for a fuller picture.
Can you fix crawl and indexing on my site?
Yes. Jamie Freeman handles technical SEO — robots, sitemaps, redirects, meta and WordPress crawl hygiene — as part of UK SEO and web development services. Use the free checker first, then book a call if you need the fix shipped.
Need crawl, indexing and SEO fixed?
Technical SEO for UK sites — robots, sitemaps, redirects, meta and WordPress crawl health handled end to end.
Explore more
Related tools & services
Jump to another free checker or book help if you want it fixed for you.
Free tools
Services