How many addresses should a sitemap contain?

As many as you have indexable pages. That sounds obvious, but it is the fastest check there is: count the addresses in your sitemap and put that beside the number of pages you think you have. If there is a factor between them, you are looking at a different file from the one you think, or at a generator that skips half your site.

Is a Disallow in robots.txt protection?

No, and that distinction matters. A Disallow is a request: well-behaved search engines honour it, the rest ignore it, and the file is readable by anyone. Whatever genuinely needs shielding belongs behind a login or behind a block on the web server. You use robots.txt to keep useful but pointless addresses out of an index, not to keep something secret.

Should you use changefreq and priority?

No. Google said years ago that it ignores both, and most other readers do the same. They are often still there because an old plugin wrote them. Leave them out: they are extra fields to keep in sync that deliver nothing, and a wrong priority suggests you are promising something you do not deliver.

A sitemap and robots.txt that also work for AI search

The sitemap is the dullest file on your site and often the only one nobody has opened since launch. What we found, and what you can check yourself.

The sitemap is the dullest file on your website. It is also often the only file nobody has opened since launch.

On a recent project I did open it, at the client's request, because they suspected something was off. It contained six addresses. The site had hundreds of pages.

There were three sitemaps, and the announced one was wrong

What sat there was not the file the system generates. It was a real file in the web root, left over from an earlier setup, and the web server serves a file like that before the request ever reaches the site.

It got better. The robots.txt announced a third file, from a plugin that was no longer installed. So there were three sitemaps in circulation: the announced one, the found one and the real one. The first two existed out of habit alone.

If you take one thing from this article: type your own sitemap address into your browser and look at what is there. Not what your admin screen says it generates, but what your domain actually serves.

The blank line that opened up your application

A scanning tool then reported something about the Disallow rules in robots.txt. Reports like that are often noise, but this one was right, and not for everybody.

Google, Bing and every reader following the current standard see a group start at a user-agent line and then take the most specific rule that matches. Those read the file correctly. But the original 1994 convention ends a group at a blank line, and the simpler crawlers and audit tools still do that. Python's standard library is one of them, and it sits underneath a fair share of the scanners out there.

The file had a blank line directly under the user-agent. For those readers the group was empty, and every Disallow below it hung on no bot at all. On top of that, those same readers take the first rule that matches, and the first rule was an Allow for the whole site. Dead twice over.

The result: for that part of the readership the customer portal, the unsubscribe links and the entire admin environment were open to crawl. Not unsafe, because there is a login on it, but exactly what you do not want in an index.

The fix is undramatic. Within a group you separate the sections with comment lines instead of blank lines, and you order the rules from specific to general. The file now sits in version control, in one form that is copied unchanged to every domain.

The sitemap again, and why on identity

Then the sitemap itself. It is now built in two steps: collect per language, then write out.

The key is the identity of an item, not its address. That is the detail a multilingual sitemap breaks on. A product is called something different in Dutch than in German, so the address differs per language while it is the same product. Key on the address and you see four separate pages. Key on the identity and you see one item in four languages.

Every address now carries the other languages with it as alternates, with the source language as the fallback for visitors from a country you do not serve. Only the languages in which the item genuinely exists, complete and reciprocal. And it is the same list as the lines in the head of the page itself, so the two can no longer drift apart.

With four domains that matters more than it sounds. The sitemap is then the only place where you can read in one go which addresses are each other's translation. Google and the AI search engines take that straight from the file, rather than fetching four pages and working it out themselves.

Four holes, and what they cost

Rebuilding it surfaced four things none of the people involved had expected.

An ordinary indexable page was in no sitemap at all, while the footer and two sidebar cards linked to it. That is a page you consider important yourself and that was never offered to your search engine.

A language without its own domain fell out of the sitemaps entirely. A language like that lives on the base address, but the file for that address served only its own language. A complete language version that was never submitted anywhere.

A noindex setting on a term was ignored. The page itself did set a noindex, so the sitemap submitted an address the page then refuses to have indexed. Two signals contradicting each other is worse than one clear signal.

And a page with a canonical pointing elsewhere was listed under its own address. That is the same mistake mirrored: you are asking for indexing of a page that says itself it belongs somewhere else.

No addresses that lead nowhere

The last rule added: an address does not belong in the sitemap if the web server catches the first part of that path itself, or if robots.txt closes it off.

That rule produced a find straight away. The newsletter signup page has a path in English and German that collides with the path of the newsletter module. The web server routes that request to the module, so those two pages cannot be opened. Not by a crawler, and not by a visitor.

That is how a sitemap check becomes a way to find broken pages. Two pages that showed up as neatly published in every overview turned out to be unreachable.

Why this weighs more for AI search

A classic search engine crawls a lot and corrects as it goes. An answer engine crawls less, reads structure more literally and gives one answer rather than ten links.

That makes mistakes more expensive. A language version that was never submitted does not exist for a machine like that. An hreflang set pointing only one way gives it no reason to connect your German page to a German question. And an address that does not open costs you not one visitor but the whole subject.

What you can check yourself in ten minutes

Five things, no tooling needed.

Open your sitemap address in your browser and count the entries. Check which sitemap robots.txt announces and whether that is the same one. Search robots.txt for blank lines inside a group. Take one page that exists in several languages and check that the references point at each other from both sides. And click three random addresses from your sitemap to see whether they open.

If any of that turns something up, it is usually not a small thing.

Three sitemaps in circulation, and the announced one was wrong