// Indexing

Is your sitemap listing pages that still exist?

Enter a domain or a sitemap address. This reads the file the way a crawler does, checks the XML, and opens some of the URLs inside to see whether they load. No signup, no email required.

// What gets checked

Every test, explained

A sitemap fails quietly. Crawlers drop what they cannot use and report nothing, so the file keeps looking correct. Here is what each test looks at.

Finding the sitemap, when you have not named one

Enter a full address and that exact file gets checked. Enter a bare domain and the tool goes looking, because sitemaps do not all live in the same place.

It reads robots.txt first, since that is where a site declares where its sitemap is. Failing that it tries /sitemap.xml and then /sitemap_index.xml. Up to three addresses in total, each followed through as many as five redirects.

That order matters. WordPress publishes at /wp-sitemap.xml and Yoast at /sitemap_index.xml, so a checker that only ever tries /sitemap.xml will tell a lot of people they have no sitemap when they do.

Whether what comes back is a sitemap at all

The status code gets checked first. A 404 means the file is not there. A 500 or a 403 means something worse, because the address exists and the server is failing on it — usually a generator timing out on a site large enough to need one.

Then the body gets looked at, not just the header. Plenty of sites answer a sitemap address with a 200 and send back their homepage or a styled 404 page. Anything checking only the status code calls that a pass. It is not one, and it usually means every made-up address on the site returns a page too.

Compressed sitemaps are handled. A .xml.gz is a normal way to publish one, and it gets unpacked before anything else happens.

Whether the XML actually parses

This is the check with no middle ground. XML either parses or it does not, and a file that does not is discarded whole. Not the broken line, not the broken half — all of it.

Two causes account for most of it. The first is an unescaped ampersand: a URL with a query string in it breaks the document at that line unless the & is written as &. The second is output printed before the XML declaration, which is fatal even when it is a single blank line.

The tool reports the parser error with the line number it came from, so there is somewhere to look rather than a verdict to argue with.

Whether it is the right kind of document

A sitemap opens with <urlset>. An index of sitemaps opens with <sitemapindex>. Anything else parses as valid XML and means nothing to a crawler, which then discards it.

The namespace gets checked too. There is exactly one correct value, http://www.sitemaps.org/schemas/sitemap/0.9, and a file missing it is rejected by validators outright. Google is often tolerant about this, which is precisely the problem: the file works until the day it does not, and nothing tells you why.

Encoding is the last of it. Sitemaps have to be UTF-8. Anything else affects the URLs with accented or non-Latin characters in them, which are exactly the addresses a crawler cannot guess at.

The URLs inside the file

Every entry needs a <loc>, and it has to be a complete address with the scheme and the hostname on it. A path on its own is not resolved against anything. The entry is dropped, silently, and the file still validates.

Then the addresses get compared against the site serving them. A sitemap may only list URLs on its own domain, so anything pointing somewhere else is ignored unless that domain is verified alongside this one.

Two softer versions of the same fault get reported separately. Listing www while serving from the bare domain sends every crawler through a redirect on every URL. So does listing http:// addresses on a site that runs on HTTPS, which is the very last thing an HTTPS migration leaves behind.

Every entry in the file is counted, not a sample of them. Duplicates are reported as information only, since crawlers deduplicate anyway.

The lastmod dates

lastmod has to be a W3C datetime: 2026-08-09, or 2026-08-09T14:30:00+02:00. Anything else is discarded, and the entry is then treated as though it had no date at all.

That costs more than it sounds like. lastmod is how a crawler decides which of your 400 pages to re-read first, so losing it means the pages you actually update get checked on the same schedule as the ones you wrote in 2019.

Dates in the future get flagged as well. Those usually mean the value is generated rather than read from the content, and a sitemap that keeps claiming the future is a sitemap whose dates stop being trusted.

changefreq and priority are not checked, because Google ignores both. They are not errors. They are just decoration.

Opening some of the URLs for real

This is the check the tool exists for, and it is the one no XML validator does.

Five addresses from the sitemap get requested, spread across the file rather than taken from the front — the first few entries are almost always the home page and the top-level sections, which are the URLs least likely to be broken.

Three things get reported. Entries that return a 404 or a server error, because a sitemap is a statement that these pages exist. Entries that redirect, because a sitemap should list the address you want indexed rather than one that forwards to it. And entries carrying a noindex, either as a meta tag or an X-Robots-Tag header, because that is the file and the page contradicting each other.

It is a sample, and the tool says so. Five failures out of five means the real number is far higher.

Sitemap index files

Large sites split into several files behind an index, and that is the correct way to do it. One file may hold 50,000 URLs and 50 MB uncompressed, and a file over either limit is rejected rather than trimmed.

When the address turns out to be an index, the first two sitemaps it lists get requested. A child that does not load takes every URL inside it out of circulation, and nothing about the index itself looks wrong — which is what a half-finished regeneration leaves behind.

The first child is read in full, so the URL checks and the live sample run against real pages rather than against a list of filenames.

One level of nesting is all the specification allows. An index pointing at another index is not followed, so everything below the second one is invisible.

// Common problems

What usually goes wrong, and how it gets fixed

Most sitemap problems come down to the same handful of causes. Here they are, with the fix.

Something is printed before the XML starts

The most common way a WordPress sitemap breaks, and the hardest to see. A plugin or theme file has a blank line after its closing PHP tag, or something is printing a notice, and that output lands in front of the XML declaration.

One blank line is enough. The document is unparseable, and the browser often renders it as something that looks almost right.

# What the file looks like on the wire

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

# The parser stops at line 1. Nothing below it is read.

# Find it with curl, where the blank line is impossible to miss:
curl -s https://example.com/sitemap.xml | head -c 200 | xxd | head -3

Then find the file printing it. Deactivate plugins one at a time if you have to, but check the theme’s functions.php first — a trailing newline after ?> at the end of that file is the classic cause. The fix is to delete the closing tag entirely, which is what the WordPress coding standards ask for anyway.

An ampersand in a URL breaks the whole file

A page with a query string gets written into the sitemap exactly as it appears in the address bar. In XML that is a syntax error, and it takes the document with it.

<!-- Breaks the file at this line -->
<url><loc>https://example.com/search?q=shoes&page=2</loc></url>

<!-- Correct -->
<url><loc>https://example.com/search?q=shoes&amp;page=2</loc></url>

Five characters need escaping in XML, and the ampersand is the one that shows up in URLs. If you are generating the file yourself, run every address through your language’s XML escaping function rather than writing it in by hand. And ask a second question while you are there: parameter URLs in a sitemap are usually filtered listing pages, which you may not want submitted at all.

The sitemap still has the old domain, or http://

Both are migration leftovers, and both are invisible from the front end. The site works. The pages load. The sitemap keeps handing crawlers addresses from before the move.

The cause is nearly always the same: the sitemap is generated from the site URL stored in the database, and that value was never updated.

# WordPress, over WP-CLI. Dry run first, always.
wp search-replace 'http://old-domain.com' 'https://example.com' --dry-run
wp search-replace 'http://old-domain.com' 'https://example.com'

# Then confirm the two that actually drive the sitemap
wp option get home
wp option get siteurl

# And regenerate. A cached sitemap will keep serving the old addresses.

Do the database first and the sitemap second, in that order. Regenerating before the URLs are corrected just rewrites the same wrong addresses. Afterwards, resubmit the sitemap in Search Console rather than waiting — it forces a read instead of leaving it to the recrawl schedule.

The sitemap lists pages that are gone

A sitemap is a statement that these pages exist and are worth indexing. Dead entries in it spend crawl budget on nothing, and they come back as errors against your site in Search Console.

This is what an unattended sitemap looks like after a year: products discontinued, posts deleted, a section restructured, and a generator that never noticed.

# Pull every URL out of the sitemap and check what each one returns
curl -s https://example.com/sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | while read -r u; do
      printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"
    done \
  | grep -v "^200"

That runs one request per URL, so point it at your own site and not at anyone else’s. What it gives you is the full list rather than the sample this page checks. Then fix the source: if the page moved, the sitemap should carry the new address, and the old one should redirect. If it is gone for good, it should be out of the sitemap and returning 410 rather than 404.

The sitemap asks for pages that are marked noindex

The file says index this. The page says do not. Google trusts the page, so the entry is wasted, and enough of them make the whole sitemap read as unreliable.

It happens when two systems disagree. An SEO plugin is set to exclude a section from the index, while the sitemap is generated from a different list that knows nothing about that setting.

# Check one page for both places a noindex can live
curl -sI https://example.com/page/ | grep -i x-robots-tag
curl -s  https://example.com/page/ | grep -i '<meta[^>]*robots'

# Either of these keeps the page out. Only one needs to be there.
<meta name="robots" content="noindex, follow">
X-Robots-Tag: noindex

Decide which one you meant, then make the other agree. If the page should be indexed, remove the noindex. If it should not, take it out of the sitemap and leave the noindex where it is — and do not block the URL in robots.txt as well, because a blocked page is never read and the noindex on it is never seen. The Robots.txt Checker will tell you whether you are doing that.

Found something broken?

Send me the result. I'll tell you what it takes to fix it, and whether it's worth paying anyone to do — including me.

Send me a message ↓

Prefer to talk? Book a free call ↗ · Or hire me on Upwork ↗ · Typical reply within one business day.

// FAQ

Questions

Do I actually need an XML sitemap?

Less than people think, and more than nothing. Google finds pages by following links, so a small site with clean internal linking gets indexed perfectly well without one. A sitemap earns its place when links alone are not enough: a large site, a new site with few inbound links, pages that are only reachable through a search form, or a site where things change faster than a crawler would notice on its own. What a sitemap never does is force indexing. It is a suggestion about what to look at, not an instruction about what to keep.

Why does this open some of the URLs instead of just checking the XML?

Because the XML being valid tells you almost nothing about whether the sitemap is any good. The most common fault by a distance is a file that parses beautifully and lists pages that were deleted a year ago, and no amount of schema checking will ever surface that. So this tool opens five of the addresses and reports what came back. It is a sample rather than the whole file, which is stated on the result, and a sample failing is enough to know the file needs attention.

My sitemap validates elsewhere but Google says it cannot read it. Why?

Usually because the two are not fetching the same thing. A validator that takes pasted XML checks what you pasted. Google checks what your server sends, and those differ whenever something is printed before the XML declaration, when a caching layer or a security module answers instead of the application, or when the file is generated on request and fails under load. That is why this tool requests the address rather than accepting a paste. Whatever it reports is what came off your server, not what is supposed to be there.

Should the sitemap list every page on the site?

No. It should list the pages you want indexed, which is a shorter list. Leave out anything carrying a noindex, anything blocked in robots.txt, tag and filter archives that duplicate content you already have, and pages that redirect. Each of those either wastes a crawl or sends a mixed signal. The test is simple enough: if you would not be happy to see the page in search results, it does not belong in the sitemap.

Does lastmod matter, or is it decoration like priority?

It matters, on one condition: it has to be accurate. Google uses lastmod to decide which pages to re-read first, and it stops using it entirely on sites where the value is obviously generated rather than real. A sitemap where every page was modified this morning is telling a crawler nothing. changefreq and priority are genuinely decoration, though. Google ignores both, and has said so plainly, so this tool does not check them.

How big can one sitemap be?

Fifty thousand URLs and fifty megabytes uncompressed, whichever comes first. Past either limit the file is rejected rather than trimmed, so the pages past the line are not the only ones affected. The answer is to split into several files and list them in a sitemap index, which every generator supports. This tool reads the first four megabytes of a sitemap, which covers most files completely. When a file is larger than that, the report says so rather than pretending it read the whole thing.

Do I need the Sitemap line in robots.txt if I submitted it in Search Console?

Not for Google, no. Submitting it there does the same job. The line in robots.txt is the version every other crawler can see, though, and it costs one line, so there is not much of an argument against having both. Whether that line is correct is a separate question from whether the sitemap is, and the Robots.txt Checker covers it — the directive has to be a full URL, which is the one place in robots.txt where a path on its own does not work.

Do you store the addresses I check?

Not beyond a cache. Results are cached for thirty minutes, so a shared link does not make the same requests again for every reader, and after that the entry expires. There is no account, no email field and no signup. Being straight about the rest: the address you check is part of the result page URL, so like any page address it appears in this server's access log and in Google Analytics. It is not published anywhere and it is never used to contact you. Scanning is limited to ten checks an hour per visitor and three an hour per domain, which is what stops the tool being pointed at somebody else's site over and over. It requests robots.txt, the sitemap, up to two files listed inside an index, and five of the URLs the sitemap names, on ports 80 and 443 only. It never submits a form.