XML Sitemap & Robots.txt: Beginner's Guide

If you're new to SEO, "sitemap" and "robots.txt" are two of the first technical terms you'll run into, and two of the easiest to get wrong. Both files quietly control how Google finds and indexes your site. Here's what they actually do and how to set them up correctly.

What Is an XML Sitemap?

An XML sitemap is a file that lists every important URL on your website, helping search engines discover and understand your content faster, especially useful for large sites, new sites with few backlinks, or pages that aren't well linked internally.

A basic sitemap entry looks like this:

<url>
  <loc>https://example.com/page</loc>
  <lastmod>2026-08-01</lastmod>
</url>

What Should Be In Your Sitemap

• All important, indexable pages: homepage, category pages, blog posts, product pages
• Only canonical URLs (the "master" version of each page)

What Should NOT Be In Your Sitemap

• Pages blocked by `robots.txt` or marked `noindex`
• Duplicate or parameter-based URLs
• Login pages, admin pages, thank-you pages, internal search result pages

How to Create and Submit a Sitemap

1. 1. Most CMS platforms (WordPress, Shopify, Wix) generate one automatically, usually at `yourdomain.com/sitemap.xml`.
2. 2. If yours doesn't, use an SEO plugin (like Yoast or Rank Math for WordPress) to generate one.
3. 3. Submit it in Google Search Console under Sitemaps, this tells Google exactly where to find it and speeds up discovery.
4. 4. Also reference your sitemap location inside your `robots.txt` file (see below).

What Is Robots.txt?

`robots.txt` is a simple text file at the root of your domain (`yourdomain.com/robots.txt`) that tells search engine crawlers which parts of your site they're allowed to crawl.

Important distinction: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results (without content) if other sites link to it: to fully keep a page out of search results, use a `noindex` meta tag instead.

Basic Robots.txt Example

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Allow: /

Sitemap: https://example.com/sitemap.xml

What to Typically Block

• Admin/login areas
• Cart and checkout pages
• Internal search result pages
• Staging or duplicate content sections

What NOT to Block

• CSS and JavaScript files: Google needs these to properly render and understand your pages
• Any page you actually want to rank

Common Mistakes to Avoid

• Blocking your entire site by accident: a stray `Disallow: /` in robots.txt can deindex your whole website. Always double-check after any site migration or redesign.
• Sitemap includes blocked/noindex pages, this sends mixed signals to Google and wastes crawl budget.
• Forgetting to update the sitemap: most CMS platforms auto-update it, but if yours is static, remember to regenerate it when you add or remove pages.
• Not submitting the sitemap to Search Console: Google can find your site without it, but submission speeds up discovery significantly, especially for new sites.

Quick Setup Checklist

• XML sitemap exists and includes only canonical, indexable pages
• Sitemap submitted in Google Search Console
• Robots.txt exists at the root domain
• Sitemap URL referenced inside robots.txt
• Admin/cart/checkout pages blocked from crawling
• CSS/JS files NOT blocked
• Verified robots.txt isn't accidentally blocking the whole site

Final Thought

Neither of these files will directly boost your rankings, but getting them wrong can quietly prevent your best content from ever being found. Set them up once, check them after every major site change, and they'll keep working in the background without needing much attention.

Frequently Asked Questions