// Crawling

What is your robots.txt actually telling Google?

Enter a domain. This reads the file the way a crawler does, shows you every rule it found, and tells you which ones are doing nothing. No signup, no email required.

// What gets checked

Every test, explained

A robots.txt fails quietly. Nothing errors, nothing warns, and the file still looks correct. Here is what each test looks at.

Whether the file is served at all

The tool asks for /robots.txt and follows up to five redirects, which is as far as Google goes before it gives up. Then it reads the status code that comes back.

Here is the part that surprises people. A crawler that cannot fetch robots.txt does not shrug and carry on. A 404 means "no rules, help yourself", which is fine. But a 500 or a 429 means "assume everything is forbidden", and Google pauses crawling of the whole site until it clears.

So a missing robots.txt is reported as information. A robots.txt returning a server error is reported as high, because it is doing more damage than not having one at all.

Whether what comes back is really a robots.txt

Plenty of sites answer /robots.txt with a 200 and then send back their homepage, or their styled 404 page. Anything checking only the status code calls that a pass.

It is not. There is no robots.txt there, and every rule you think you wrote is absent. Worse, it usually means the 404 handler returns 200 for everything, so every made-up address on the site returns a page too.

The tool looks at the body, not just the header. HTML gets reported. A text file with an unusual content type does not, because plenty of correct servers label a .txt as application/octet-stream and warning about that would be wrong more often than right.

Rules that close the whole site

One line does it: Disallow: / under User-agent: *. That covers every page on the domain.

This is the finding the tool exists for. It is nearly always a staging robots.txt that went live with the site and was never changed back, and on WordPress it is nearly always one checkbox in Settings. Nothing breaks visibly, so it gets noticed weeks later as traffic falling away rather than as a configuration problem.

A group naming Google or Bing specifically gets the same treatment, because a named group overrides the general one. Open rules further up the file do not soften it.

One thing the tool will not do is read Disallow: with nothing after it as a block. Empty means the opposite — allow everything — and getting that backwards would put a critical finding on a perfectly healthy site.

Rules that no crawler reads

A robots.txt is read in groups. Every group opens with a User-agent line saying who it is for. Allow and Disallow lines written above the first one belong to nobody, and every crawler skips them.

Typos work the same way. Dissallow is not a broken rule, it is an absent one, and nothing anywhere reports an error. The file still parses. It just does less than it looks like it does.

The tool also flags Noindex: and Nofollow:. Google honoured those here unofficially for years, then stopped in September 2019. Anyone still relying on them has pages they believe are hidden and are not.

Directives that are spelled correctly but simply unused, like Crawl-delay, are reported as information rather than as errors. Bing and Yandex do act on that one. Google ignores it.

The files a page needs in order to render

Google renders your pages before it judges them, using the same CSS, JavaScript and images a visitor gets. So blocking /wp-content/ or /assets/ does not hide the page. It hides how the page looks.

The renderer then sees unstyled text with the layout missing, and scores what it can see. On a mobile-first index that costs more than it used to.

Blocking asset folders was standard advice once, and it is still sitting in a lot of older files. Google has been telling people not to do it since 2014.

The Sitemap lines, and whether they lead anywhere

The Sitemap directive is the one line in robots.txt that has to be a full URL, scheme and hostname included. A path on its own gets dropped without comment.

The tool requests the first two declared sitemaps and reports the status. A sitemap that 404s is common after a migration or after a plugin gets swapped out, and nothing breaks visibly when it happens. New pages just take longer than they should to be found.

Whether the XML inside is valid is a separate question. This tool asks only whether the address answers.

www and the apex — two origins, two files

A robots.txt governs exactly one origin. So example.com and www.example.com each read their own, and a crawler applies whichever one matches the hostname it asked for.

Both get requested in practice, whichever form you consider canonical. When the two files disagree, which rules apply depends on how the crawler arrived.

This hides well, because whichever one you check looks correct. Redirecting one hostname to the other fixes it permanently, since then there is only one file. Whether that redirect exists is the Redirect Chain Checker’s job.

// Common problems

What usually goes wrong, and how it gets fixed

Most robots.txt problems come down to the same handful of causes. Here they are, with the fix.

The site went live with the staging robots.txt

The most common critical result, and on WordPress it is usually not a file anyone edited. Settings, Reading, "Discourage search engines from indexing this site" writes exactly this:

# What the checkbox produces
User-agent: *
Disallow: /

# What you want instead
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Untick the box first. Then check whether a real robots.txt file exists in the webroot, because a physical file overrides the virtual one WordPress generates — and if one is sitting there with Disallow: / in it, unticking the box changes nothing. That is the step people miss. Afterwards, request reindexing in Search Console rather than waiting, because recovery is much slower than the drop was.

The rules sit above the first User-agent line

Someone added a rule at the top of the file, where it reads naturally and does nothing. Every crawler skips it.

# Wrong. These two belong to no group.
Disallow: /private/
Disallow: /tmp/

User-agent: *
Disallow: /admin/

# Right
User-agent: *
Disallow: /private/
Disallow: /tmp/
Disallow: /admin/

The order matters, not the indentation. A group runs from its User-agent line until the next one, and blank lines between rules are ignored. So anything above the first User-agent is orphaned no matter how it is laid out.

Noindex is being used to hide a page

It worked once. Google dropped support in September 2019, and the lines have been inert ever since. The pages are eligible to appear in search results and nothing says so.

There is a second trap here. Blocking a page in robots.txt does not remove it from the index either — it stops the page being read, which is not the same thing. A blocked page with links pointing at it can still be listed, showing no description. To keep a page out you have to let the crawler reach it and find a noindex on the page itself:

# In robots.txt: delete the Noindex line and leave the page crawlable

# In the page head
<meta name="robots" content="noindex, follow">

# Or as a response header, for PDFs and anything not HTML
Header set X-Robots-Tag "noindex"

Those two are mutually exclusive with a Disallow. Block the URL and the crawler never arrives, never reads the tag, and the page stays exactly where it is. Allow it, wait for the recrawl, and only then block it if you also want to save the crawl budget.

The theme files are blocked

An old hardening guide, or a security plugin with an opinion. The intent was to keep crawlers out of the WordPress internals, and the effect is that Google cannot see what your pages look like.

# Blocks the CSS and JavaScript needed to render the page
User-agent: *
Disallow: /wp-content/
Disallow: /wp-includes/

# Block the admin instead, and leave the theme alone
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The admin-ajax.php exception matters, because a lot of themes and plugins load content through it. Blocking it breaks rendering on exactly the pages that rely on it most. Check the result in the URL Inspection tool in Search Console afterwards — it shows you the rendered screenshot, which is the only way to see this properly.

The sitemap line is relative, or points at a file that is gone

Two different mistakes with the same outcome. Either way the sitemap is not being found through robots.txt.

# Ignored. This directive is not like the others.
Sitemap: /sitemap.xml

# Correct
Sitemap: https://example.com/sitemap_index.xml

Allow and Disallow take paths. Sitemap takes a full URL, and it sits outside the groups, so it applies no matter where in the file you put it. Once it is right, open the URL yourself and confirm it returns a 200 rather than a redirect to the homepage, which is what a removed sitemap usually does.

Found something broken?

Send me the result. I'll tell you what it takes to fix it, and whether it's worth paying anyone to do — including me.

Send me a message ↓

Prefer to talk? Book a free call ↗ · Or hire me on Upwork ↗ · Typical reply within one business day.

// FAQ

Questions

Does blocking a page in robots.txt keep it out of Google?

No, and this is the most expensive misunderstanding about this file. Disallow stops a crawler reading the page. It does not stop the URL being listed. If other sites link to it, Google can index the address without ever fetching it, and you get a result with no description. Worse, because the page was never read, any noindex tag on it is invisible. So to remove a page you have to allow the crawler in, let it see the noindex, and only block the URL afterwards if you also want to save the crawl budget. Use robots.txt to manage crawling. Use noindex to manage indexing. They are different jobs.

I have no robots.txt at all. Is that a problem?

Not in itself. No file means no rules, and crawlers treat that as permission to crawl everything, which is usually what a small site wants anyway. The tool reports it as information rather than a fault. It is still worth adding one, mostly so you have somewhere to declare your sitemap and somewhere to keep search result pages and filtered listings out of the crawl. Just do not add one purely to have one, because an empty file and no file mean exactly the same thing.

Why is a server error on robots.txt worse than a missing one?

Because crawlers read the two differently. A 404 says there are no rules, so everything is fair game. A 500 or a 429 says the answer is temporarily unavailable, and Google plays it safe by assuming everything is forbidden until the file comes back. Crawling of the whole site pauses. Pages already indexed hold for a while, then start dropping out. On a static text file this nearly always means the request is reaching the application instead of the filesystem, or a rewrite rule is catching it.

Does Noindex work in robots.txt?

It did, unofficially, and Google stopped supporting it in September 2019. The lines are ignored now. Nothing errors and the file still validates, which is why so many of them are still sitting in production files doing nothing. If you have pages you believe are hidden by a Noindex line in robots.txt, they are not hidden. Move the directive onto the page as a meta tag, or send it as an X-Robots-Tag header for files like PDFs that have no head section.

Should I block /wp-admin/?

Yes, with one exception. Disallow the directory and then explicitly allow admin-ajax.php, because a lot of themes and plugins load real page content through it and blocking that breaks how those pages render for Google. What you should not block is wp-content or wp-includes. That is where your CSS, JavaScript and images live, and Google renders the page before scoring it. Blocking them does not hide anything. It just makes your site look broken to the one visitor whose opinion you were trying to influence.

Why does it check www when I only entered the apex?

Because a robots.txt only governs the exact origin it is served from, so www and the apex are two separate files. Both get requested by crawlers in practice, whichever form you treat as canonical. They disagree more often than people expect, usually because one was edited and the other was forgotten, and it hides well because whichever one you check looks correct. Both are covered in a single scan, which is also why it only counts once against the rate limit.

Do you store the addresses I check?

Not beyond a cache. Results are cached for thirty minutes, so a shared link does not make the same requests again for every reader, and after that the entry expires. There is no account, no email field and no signup. Being straight about the rest: the address you check is part of the result page URL, so like any page address it appears in this server's access log and in Google Analytics. It is not published anywhere and it is never used to contact you. Scanning is limited to ten checks an hour per visitor and three an hour per domain, which is what stops the tool being pointed at somebody else's site over and over. It requests one file, on ports 80 and 443 only, and it never submits a form.