robots.txt and Sitemap Generator

Enter the user agent, allowed and blocked paths, and sitemap URLs. Add the page URLs yourself to generate the XML sitemap.

Runs locally in your browser
This tool processes all data locally in your browser.
InputUse one path or URL per line. This generator validates syntax locally; it does not fetch a site.

How to build robots.txt and sitemap.xml

Fill the five fields and press Build files: robots.txt is written from the user-agent, Disallow, Allow and Sitemap lines, and sitemap.xml from the URL list. Both are produced in this browser; no site is fetched and nothing is uploaded.

Only absolute http and https addresses are written into the sitemap. Anything else is left out of the file and listed under the fields, so a malformed entry never reaches your site.

  1. Enter the crawler name in User-agent, or leave the default * for all crawlers. Several names can be separated by commas or written on their own lines; each becomes its own User-agent line.
  2. List blocked paths under Disallow and exceptions under Allow, one per line. A path without a leading slash, or a full URL, is rewritten to a path and reported.
  3. Add the address of each sitemap you publish under Sitemap URLs, and paste the page addresses for sitemap.xml in the last field, one per line.
  4. Press Build files. Both panes fill at once, together with a count of the lines that were changed, skipped or repeated.
  5. Copy or download each file and publish it at the site root: /robots.txt and /sitemap.xml.

What the files do, what the builder changes and where its limits are

What the two files do

robots.txt tells crawlers which paths they may request. It is a request, not a security boundary: a crawler that ignores it can still fetch the page, so anything that must stay private belongs behind authentication. A rule belongs to the user-agent group above it, which is why the generated file starts with User-agent and then lists the paths.

sitemap.xml lists the addresses you want indexed. It helps discovery rather than ranking, and a URL blocked in robots.txt will not be indexed even when it also appears in the sitemap.

What the builder changes in your input

Paths are corrected so the file stays valid: a missing leading slash is added, a pasted full URL becomes its path and query, and a line that still cannot be a path is left out of the file and named in the status. Lines starting with # are kept as comments, which robots.txt allows. Repeated URL lines and repeated Sitemap lines are written once and reported as duplicates. Wildcards (*) and the end anchor ($) pass through untouched.

What this page does not do

There is no Crawl-delay, no separate group per crawler, and no lastmod, changefreq or priority in the sitemap. One sitemap file holds at most 50,000 URLs, and the page says so instead of writing more. Nothing is fetched, so an address that no longer exists is still written exactly as typed; the page cannot know whether a URL answers 200, redirects or returns 404.

Recent tools: