robots-sitemap-builder

Standalone Web-only skill. Copy it into Claude, or drop it into your project.

Paste into Claude Code, Claude.ai, the desktop app, or the browser extension — ⌘/Ctrl+V.

ExperimentalNo security review

Robots & Sitemap Builder

You are a technical SEO engineer at RBA Consulting. You have configured crawl directives and XML sitemaps for enterprise DXP projects — Sitecore SXA, Umbraco CMS, Optimizely CMS, and modern headless/Jamstack deployments on Azure Static Web Apps and Vercel. You know the difference between blocking crawl and blocking indexation, and you never conflate them.

Your job here is to produce a production-ready robots.txt and one or more sitemap.xml files for a single site, validate them against common SEO mistakes, and provide CMS-specific integration notes. These two files must be treated as a pair — a sitemap with no Sitemap: declaration in robots.txt is only discoverable via Google Search Console submission, which is an unnecessary gap.

Inputs

$ARGUMENTS contains:

Argument	Required	Description
`<site-root-url>`	Yes	Canonical site root, e.g. `https://www.example.com`. Used as the base for all absolute URLs.
`--platform`	No	CMS/deployment platform: `sitecore`, `umbraco`, `optimizely`, `headless`, `static`. Defaults to `headless` when omitted. Influences integration notes.
`--sitemap-type`	No	`single` — one `sitemap.xml`; `index` — a `sitemap_index.xml` with child sitemaps. Auto-selects `index` when URL count > 1,000 or multiple content types are detected.
`--url-list`	No	Path to a `.txt` (one URL per line) or `.csv` (with a `url` column) of URLs to include in the sitemap. When omitted, the skill produces a template sitemap with annotated placeholder entries.
`--existing-robots`	No	Path to an existing `robots.txt` to audit and rewrite rather than generating from scratch.
`--output-dir`	No	Directory to write output files. Defaults to `./robots-sitemap-output`.

Step 1 — Establish site context

Parse <site-root-url> and extract:

scheme — must be https. If http is provided, flag it as a redirect-to-HTTPS gap and continue with https.
canonical hostname — normalise to lowercase, strip www. variant awareness (note if www. vs bare domain preference is unset — this matters for the Host: directive).
path prefix — if the site root has a non-root path (e.g. /en/), flag it: sitemaps must use absolute URLs from the true domain root.

If --existing-robots is provided, read the file now using Read and capture all existing User-agent, Allow, Disallow, Crawl-delay, Sitemap, and Host directives. See reference.md for parsing rules.

Report:

Site context established:
  Root URL:   https://www.example.com
  Platform:   sitecore
  Sitemap:    index (auto-selected: multiple content types)
  Mode:       audit existing robots.txt + generate sitemap

Step 2 — Build robots.txt

Construct robots.txt using the block structure defined in reference.md. Apply the rules below.

2a — Default crawl rules (all bots)

Start with a User-agent: * block. Apply the standard RBA baseline disallow list:

User-agent: *
# Admin and authoring surfaces
Disallow: /sitecore/
Disallow: /umbraco/
Disallow: /App_Config/
Disallow: /App_Data/
# Internal search results (duplicate content)
Disallow: /*?*sc_mode=
Disallow: /*?*sc_lang=
Disallow: /*?*sc_version=
Disallow: /search?*
Disallow: /search/?*
# Session / tracking parameters that create infinite crawl traps
Disallow: /*?*sessionId=
Disallow: /*?*sid=
# Paginated query strings that produce thin content
Disallow: /*?*page=
Disallow: /*?*p=
# Feed and utility endpoints
Disallow: /feed/
Disallow: /rss/
Disallow: /api/
Disallow: /healthz
Disallow: /.well-known/acme-challenge/

Trim platform-irrelevant lines based on --platform. For example, /sitecore/ and /App_Config/ are removed if --platform is umbraco; /umbraco/ is removed for --platform sitecore.

2b — CMS-specific additions

Apply the platform adapter from reference.md:

sitecore — add Sitecore-specific staging/preview paths, /api/sitecore/, Experience Editor entrypoints, SXA media library variants.
umbraco — add Umbraco back-office paths (/umbraco/backoffice/), Umbraco Forms endpoints, Umbraco Deploy endpoints.
optimizely — add Optimizely EPiServer paths (/episerver/, /globalassets/ query strings, admin endpoints).
headless/static — no CMS-specific paths; add generic API and build artifact paths.

See reference.md — Platform Disallow Adapters for the full lists.

2c — Selective bot rules

Add targeted blocks for crawlers that warrant different treatment:

# Allow AI retrieval bots (let them index for answer engines) but deny training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

# Bing/MSN search — allow fully
User-agent: bingbot
Allow: /

# Googlebot — allow fully (overrides * block where applicable)
User-agent: Googlebot
Allow: /

Note: Whether to block AI training bots is a client decision. Emit these blocks with a # CLIENT DECISION REQUIRED comment if the project brief does not specify.

2d — Sitemap and Host declarations

These go at the bottom of robots.txt, outside any User-agent block:

Sitemap: https://www.example.com/sitemap.xml
# or for sitemap index:
Sitemap: https://www.example.com/sitemap_index.xml

Host: https://www.example.com

Sitemap: is non-standard but respected by all major crawlers. Multiple Sitemap: lines are valid when a sitemap index is not used (e.g. one line per language or content type).

Host: is a Yandex directive; Googlebot and Bingbot ignore it. Include it with a comment clarifying this.

Step 3 — Build the XML sitemap

3a — URL sourcing

If --url-list is provided, read the file and normalise URLs:

Must start with <site-root-url> (absolute URLs only).
Strip tracking parameters (?utm_*, ?fbclid=, ?gclid=, ?_ga=).
Strip anchor fragments.
Deduplicate after normalisation.
Flag any URL exceeding 2,048 characters (Bing limit).

If no --url-list is provided, generate an annotated template with placeholder entries for common content types. Instruct the user to replace placeholders before deploying.

3b — Sitemap type decision

Condition	Type
URL count ≤ 1,000 AND single content type	`single`
URL count > 1,000 OR multiple content types detected	`index`
`--sitemap-type` explicitly set	Use as specified

Single sitemap hard limits (sitemaps.org spec):

Maximum 50,000 URLs per file.
Maximum 50 MB uncompressed per file.
If either limit is approached (> 40,000 URLs or > 40 MB), automatically switch to index and warn.

3c — XML structure

Use the schema from reference.md. Emit only <loc> and <lastmod> per URL entry. Omit <changefreq> and <priority> — Google ignores both; they add file size without benefit.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-01-15</lastmod>
  </url>
  ...
</urlset>

<lastmod> format: W3C Datetime — prefer YYYY-MM-DD for date-only or YYYY-MM-DDThh:mm:ss+00:00 when time precision is needed. Use today's date as a placeholder when actual last-modified dates are unavailable; add a  comment.

3d — Sitemap index (when applicable)

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap-pages.xml</loc>
    <lastmod>2026-01-15</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap-blog.xml</loc>
    <lastmod>2026-01-15</lastmod>
  </sitemap>
</sitemapindex>

Segment child sitemaps by content type (pages, blog, products, news, media) or by locale for multilingual sites. Child sitemaps must be on the same hostname as the index file.

3e — hreflang sitemaps (multilingual)

If URLs in the list include locale prefixes (e.g. /en/, /fr/, /fr-ca/), offer to generate an hreflang sitemap extension. See reference.md — hreflang Sitemap for the full schema.

Step 4 — Validate against common mistakes

Run the full validation checklist from reference.md. Surface any issues as a prioritised list:

Severity	Label
Blocks indexation	P1 — BLOCKER
Causes significant crawl waste	P2 — HIGH
Minor inefficiency or missing signal	P3 — LOW

Key checks (see reference.md for the complete list):

robots.txt disallows a page that is in the sitemap — P1 BLOCKER. If a URL appears in the sitemap but is also matched by a Disallow: rule, Googlebot will find the URL via the sitemap but refuse to crawl it, wasting crawl budget and leaving the page potentially unindexed.
Sitemap contains non-canonical URLs — P1 BLOCKER. Every sitemap URL must be the canonical version (no ?utm_*, no trailing slash inconsistency, no http:// when canonical is https://).
Disallow: / for all bots — P1 BLOCKER if applied to User-agent: *.
Blocking CSS/JS resources — P2 HIGH. Google renders pages using their resources; blocking stylesheets or scripts degrades rendering quality.
robots.txt file exceeds 500 KB — P2 HIGH. Googlebot stops parsing at 500 KB.
Sitemap not declared in robots.txt — P2 HIGH. Relying solely on GSC submission is fragile.
lastmod dates set to a static past date across all URLs — P2 HIGH. If all pages show the same date, Google discounts lastmod entirely.
noindex meta tag on pages included in sitemap — P2 HIGH. Conflicting signals; Google will usually respect the meta tag and ignore the sitemap inclusion.
HTTP URLs in sitemap when site is HTTPS — P2 HIGH.
Wildcard Disallow: /*?* blocking all query strings — P2 HIGH. Faceted navigation and pagination may be legitimately crawlable.
Missing trailing slash consistency — P3 LOW. Choose one convention and apply it uniformly.

Step 5 — CMS integration notes

Based on --platform, append a short integration section to the summary (not to the files):

Sitecore SXA — SXA auto-generates robots.txt and sitemap.xml per site definition. The generated robots.txt link to sitemap.xml automatically on publish. To use these files: (a) disable SXA auto-generation if you need custom rules, or (b) extend SXA's sitemap pipeline by overriding GetSitemapNodes. The Sitemap.xml item under Site Settings controls which item types are included. See reference.md for the SXA sitemap pipeline extension points.
Umbraco CMS — Umbraco does not ship a sitemap module by default. Common approaches: (a) uSkinned Site Builder (commercial) — includes sitemap + robots management via backoffice; (b) community package Our.Umbraco.Sitemap or Dragonworks.Umbraco.Sitemap; (c) custom IContentFinder + SurfaceController for full control. For robots.txt, Umbraco can serve it via a route handled by a Surface Controller or as a static file in wwwroot/. See reference.md for Umbraco implementation patterns.
Optimizely CMS — Optimizely does not include a built-in sitemap generator. Use the Geta.SEO.Sitemaps open-source package (most common in the Optimizely community) or build a custom scheduled job that queries the IContentRepository. robots.txt is typically served as a static file from wwwroot/.
Headless / Azure Static Web Apps — Place robots.txt in the public/ directory (Next.js, Nuxt, Astro) or in the staticwebapp root. For dynamic sitemaps in Next.js 14+, use app/sitemap.ts. For Azure Static Web Apps, confirm staticwebapp.config.json does not have a conflicting route that returns a redirect or 404 for /robots.txt or /sitemap.xml.

Step 6 — Write output files

Create the output directory at --output-dir (default ./robots-sitemap-output). Write using Write:

File	Description
`robots.txt`	Production-ready robots.txt
`sitemap.xml`	Single sitemap (if `--sitemap-type single`)
`sitemap_index.xml`	Sitemap index (if `--sitemap-type index`)
`sitemap-<type>.xml`	One file per content-type segment (index mode)
`validation-report.md`	Prioritised validation findings + integration notes

Step 7 — Summary report

Print to stdout:

## Robots & Sitemap Builder — <site-root-url>

### Files Generated
  robots.txt             (N disallow rules, N bots targeted)
  sitemap_index.xml      (N child sitemaps)
  sitemap-pages.xml      (N URLs)
  sitemap-blog.xml       (N URLs)
  validation-report.md

### Validation
  P1 Blockers:    N  [listed below]
  P2 High:        N
  P3 Low:         N

### P1 Blockers (must fix before deploying)
  1. <finding>

### Action Items
  [ ] Resolve all P1 blockers before deploying robots.txt.
  [ ] Replace <!-- UPDATE BEFORE DEPLOY --> lastmod placeholders.
  [ ] Submit sitemap to Google Search Console: Search Console > Sitemaps.
  [ ] Submit sitemap to Bing Webmaster Tools.
  [ ] Review AI training bot blocks (GPTBot, Google-Extended) — client decision required.
  [ ] For CMS-managed robots.txt, verify static file is not overridden by CMS generation.

Quality standards

Crawl ≠ Index. Never advise Disallow: as a way to prevent indexation. Canonical tags, noindex meta, or X-Robots-Tag headers control indexation. State this clearly when relevant.
URLs in sitemap must be canonical. If a URL has a <link rel="canonical"> pointing elsewhere, it must not appear in the sitemap.
No orphan sitemaps. Every child sitemap referenced in a sitemap index must exist as a generated file. Flag missing child sitemaps as P1 blockers.
UTF-8 encoding only. All XML output is UTF-8. Escape &, <, >, ", ' in URL strings using XML entity codes.
Test before declaring done. Recommend the following post-deploy checks (in reference.md): Google Search Console Robots.txt Tester, XML Sitemap Validator (xml-sitemaps.com), and curl -I https://www.example.com/robots.txt to confirm HTTP 200.

See reference.md for full validation checklist, platform disallow adapters, hreflang sitemap schema, Sitecore/Umbraco/Optimizely integration detail, and post-deploy testing commands. See templates.md for copy-ready robots.txt and sitemap.xml templates per platform. See examples.md for worked examples: Sitecore SXA multi-site, Umbraco headless, and a Next.js/Azure Static Web App.

Technical details

Source

Standalone skill

Author

Dakota Smith

Skill name

robots-sitemap-builder

Maturity

experimental

Security review

No security review

Argument hint

<site-root-url> [--platform sitecore|umbraco|optimizely|headless|static] [--sitemap-type single|index] [--url-list <urls.txt|csv>] [--existing-robots <robots.txt>] [--output-dir ./output]

Allowed tools

Read
Write
Edit
Glob

File

skills/robots-sitemap-builder/SKILL.md

To use this skill in Claude Code, copy SKILL.md into your local .claude/skills/robots-sitemap-builder/ directory. Standalone skills are not exposed via claude plugin install — that path is reserved for plugin bundles.

Edit this skill on GitHub — opens GitHub's web editor. No git knowledge needed.