XML Sitemaps Guide: What to Include, What to Leave Out
An XML sitemap is a list of the URLs you'd like search engines to know about. That's it. It does not force indexing, it doesn't improve rankings directly, and Google is free to ignore any URL in it. People expect far more from sitemaps than they deliver, and then neglect the one thing that makes them useful: an accurate list.
Here's how I'd set one up and keep it honest.
When a sitemap actually helps
Google finds most pages by following links. A sitemap matters most when that isn't enough:
- New sites with few external links pointing in, so discovery is slow.
- Large sites (thousands of product or listing pages) where some pages sit deep in the structure.
- Sites with fresh content such as news or a classifieds portal, where getting new URLs seen quickly has value.
- Pages with weak internal linking, though the better fix there is the internal linking itself.
A 12-page website for a CA firm in Pune with a clean menu will be crawled fine without one. Add it anyway, since it costs nothing, but don't expect a change.
What belongs in the sitemap
Only URLs you want indexed. In practice that means each URL should:
- Return a 200 status code (no redirects, no 404s).
- Be the canonical version of itself, not a duplicate pointing elsewhere.
- Not carry a noindex tag.
- Not be blocked by robots.txt.
Leave out tag archives you've noindexed, filtered or sorted URLs, paginated duplicates, thank-you pages, login pages and internal search results. A sitemap full of redirected or noindexed URLs sends mixed signals and makes the Search Console report harder to read.
The format
A minimal sitemap looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/services/gst-filing/</loc>
<lastmod>2026-09-14</lastmod>
</url>
</urlset>
Use absolute URLs with the correct protocol and host. Google ignores the <priority> and <changefreq> fields, so don't spend time tuning them. It does use <lastmod> when it's consistently accurate, which means updating it only when the page content meaningfully changes, not on every page load or every time a sidebar widget refreshes.
Size limits and sitemap index files
A single sitemap file can hold up to 50,000 URLs and must be no larger than 50 MB uncompressed. Beyond that, split it into several files and list them in a sitemap index:
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://example.com/sitemap-products.xml</loc></sitemap>
<sitemap><loc>https://example.com/sitemap-blog.xml</loc></sitemap>
</sitemapindex>
Splitting by content type is useful even below the limit. If your store has separate sitemaps for products, categories and blog posts, the Search Console report shows indexing per file, and you can see at a glance that, say, 90% of blog posts are indexed but only a fraction of product pages are. That tells you where to look.
There are also image, video and news sitemap extensions. Image sitemaps can help if your images load through JavaScript, and news sitemaps matter for publishers in Google News. Most sites don't need them.
How to create one
- WordPress: WordPress has a built-in sitemap at
/wp-sitemap.xml, and SEO plugins like Yoast or Rank Math replace it with their own. Use one, not both. - Shopify: generates
/sitemap.xmlautomatically, split into product, collection, page and blog files. - Custom or static sites: generate it during your build, or use a crawler. For small sites the free sitemap generator produces a file you can upload.
Next.js, Astro and most modern frameworks have sitemap packages or built-in support. The important part is that it regenerates when pages are added or removed.
Submitting and reading the report
Add a Sitemap: line to your robots.txt (see the robots.txt guide) and submit the sitemap URL in Search Console under Sitemaps. The report shows whether Google could fetch the file and how many URLs it discovered.
For indexing detail, open the Pages report and filter by sitemap. Common findings:
| Message | Likely cause |
|---|---|
| Couldn't fetch | Wrong URL, server error, or blocked by robots.txt |
| Discovered, currently not indexed | Google knows the URL but hasn't crawled it yet, often a quality or crawl priority issue |
| Crawled, currently not indexed | Google saw the page and decided not to index it, usually thin or duplicate content |
| Page with redirect | You listed a URL that redirects; replace it with the destination |
"Crawled, currently not indexed" is not a sitemap problem. No sitemap tweak will fix it; the page itself needs to be more useful or more clearly distinct from similar pages. Duplicate content covers one frequent cause.
Mistakes I see often
- Listing both
httpandhttps, orwwwand non-www, versions. - Leaving a staging domain in the
<loc>URLs after migration. - Setting every
<lastmod>to today's date automatically, which teaches Google to ignore the field. - Two plugins generating competing sitemaps.
- Submitting a sitemap and assuming it guarantees indexing.
Download your current sitemap, pick twenty URLs at random and check each one: does it return 200, is it canonical, is it indexable? If even a few fail, fix the generator settings before anything else.