robots.txt checker
Find out which of your pages Google isn’t allowed to crawl — and which line of robots.txt is the reason.
- No sign-up to start
- 138 checks
- 500 pages free
robots.txt checker: what’s checked
robots.txt tells crawlers where they may not go. One over-broad line — Disallow: /blog meant for /blog-drafts, or an old rule blocking /wp-content/ — can hide whole sections from Google, or stop it loading the CSS and JavaScript it needs to see your pages properly.
This checker reads your robots.txt the way Googlebot does (its own group if there is one, otherwise *, longest matching rule wins, wildcards and $ supported) and tests it against every page the crawl found, the scripts and stylesheets those pages load, and every URL in your sitemap. Each blocked URL comes with the rule that blocks it.
14 Crawling & indexing Pages have to be crawled and indexed before they can rank at all. These checks cover the files, canonicals and directives that tell search engines what they may fetch and keep.
-
Broken canonical tag
A canonical tag pointing to a broken or wrong URL can deindex the page. Point it at the correct, reachable preferred URL.
-
Canonical tags chained through another page
Google does not follow canonical tags from one page to the next -- a chained canonical is treated as a weak hint and it picks its own winner instead, so the consolidation you set up is forfeited. Point every canonical straight at the final URL.
-
Missing canonical tag
A canonical tag tells search engines which URL is the “real” one when several are similar. Add <link rel="canonical"> to avoid duplicate-content dilution.
-
Missing robots.txt
robots.txt tells crawlers what they may access. Add one (even an allow-all) and point it at your sitemap.
-
Page blocked by robots.txt
Googlebot won’t crawl this page, so its content can’t rank — and a noindex on it is never seen. If the page should be found, remove or narrow the Disallow rule shown; if it’s meant to be private, stop linking to it and keep it out of the sitemap. Test the rule in Search Console’s robots.txt report.
-
Page set to noindex
This page tells search engines not to index it, so it can’t rank. If that’s unintended, remove the noindex directive.
-
Pages buried too deep
Pages many clicks from the homepage get crawled and ranked less. Flatten your structure and link to deep pages.
-
SEO-unfriendly URLs
Long, parameter-heavy URLs are harder for users and search engines to parse. Use short, readable, keyword-relevant paths.
-
Scripts or CSS blocked by robots.txt
Google renders pages with their CSS and JavaScript. Blocked files can make a page look broken or empty to it. Remove the Disallow rules that cover theme, plugin or asset folders; there’s no SEO reason to hide scripts and stylesheets.
-
Sitemap URLs blocked by robots.txt
The sitemap asks Google to crawl URLs that robots.txt forbids, which Search Console reports as “Submitted URL blocked by robots.txt”. Either take those URLs out of the sitemap or stop blocking them.
-
Soft 404 (server returns 200 for missing URLs)
Returning HTTP 200 for missing pages makes search engines index empty pages. Serve a real 404 status for URLs that don’t exist.
-
robots.txt doesn’t name a sitemap
Add a line like “Sitemap: https://example.com/sitemap.xml” to robots.txt. Every crawler reads robots.txt first, so this is how Bing and others find your sitemap without a manual submission.
-
robots.txt malformed
A malformed robots.txt can accidentally block your whole site. Validate it and confirm it isn’t disallowing important paths.
-
www / apex not canonicalized
Serving both www and non-www without redirecting splits your ranking signals. Pick one and 301-redirect the other.
4 Sitemap A sitemap tells crawlers which URLs you consider worth indexing. These checks make sure it exists, parses, and does not list pages that contradict it.
-
Missing sitemap.xml
An XML sitemap helps search engines discover all your pages. Generate one and reference it in robots.txt.
-
Sitemap lists broken or redirecting URLs
Your sitemap tells search engines these URLs are worth indexing, but they error or redirect. Remove the dead entries and point the rest straight at their final URL so crawl budget goes to real pages.
-
Sitemap lists pages that block indexing
These pages are listed in the sitemap as worth indexing, yet the pages themselves say otherwise via noindex or a canonical pointing elsewhere. One of the two is wrong — decide which, then make them agree.
-
sitemap.xml malformed
A malformed sitemap can be ignored by search engines. Validate and fix the XML so all URLs are discoverable.
How it works
-
1
Enter your URL
Paste your homepage. No account, no install, no verification.
-
2
We crawl your site
The crawler follows your internal links and runs every check against each page it reaches.
-
3
Get your results
See your score and issue counts at once. A free account shows every finding, page by page, with how to fix it.
Frequently asked questions
Does blocking a page in robots.txt keep it out of Google?
No. It stops Google crawling the page, not indexing the URL — a blocked page linked from elsewhere can still appear in results, without a description. To keep a page out of the index, allow crawling and use noindex; Google can’t see a noindex on a page it isn’t allowed to fetch.
Which rules does it follow?
Google’s: the group for Googlebot if your file has one, otherwise the * group; the longest matching Allow or Disallow wins, Allow wins a tie, * matches anything and $ marks the end of the URL.
Should I block CSS and JavaScript?
No. Google renders pages with their CSS and JavaScript, and blocking them can make a page look broken or empty to it. The checker flags any of your own scripts or stylesheets robots.txt blocks.
Related free tools
Ready to check your site?
It takes a couple of minutes and costs nothing.