Robots.txt Generator Create Crawl Rules for Search Engines and AI Bots

What this generator does

This tool takes a set of choices about which crawlers may access a site, which paths should stay out of the crawl, and where the sitemap lives, and returns a root-ready robots.txt file. It handles the wildcard group for unnamed bots, named groups for specific search engines and AI crawlers, path directives, an optional crawl delay, and an optional sitemap line. The generated file can be copied to the clipboard or downloaded as a plain text file, ready to upload to the root of the domain.

Everything runs inside the browser. The file is built locally from the options on the page, and nothing is sent anywhere. If the AI insight button is pressed, only an anonymous summary of the selected options, the count of blocked and allowed bots, and whether a sitemap was added is passed to a small serverless function that runs once and then stops. No domain name, no path, and no personal detail is included.

The generator is a starting point for a well-shaped robots.txt file, not a replacement for the webmaster tools of each search engine. Use it to produce a clean file, upload it, and then verify the live response through the webmaster tools of the engines that matter to the site.

How the Robots Exclusion Protocol works

The Robots Exclusion Protocol is a small convention that has been in use since the mid-1990s and was formally published as RFC 9309 in 2022. A file named robots.txt is placed at the root of a host, and a crawler that wants to request a URL first fetches that file and checks whether the path is allowed for its user agent. If the path is disallowed, the crawler is expected to skip the request. If the file is missing or returns an error, most crawlers assume that everything is allowed.

The protocol is voluntary. A well-behaved crawler such as Googlebot, Bingbot, or YandexBot follows the rules, but a scraper that wants to ignore them can simply not fetch the file. This is why robots.txt is not a security mechanism. It is a signal, and its value comes from the fact that the search engines that send most of the traffic do follow it. For genuine access control, use authentication, authorisation, or server-level rules.

The file is parsed top-down, and the rules are grouped by user agent. The special wildcard agent, written as an asterisk, applies to every crawler that does not have its own named group. A named group takes precedence over the wildcard, so a site that allows everything but wants to block one specific agent can write a wildcard group that allows and a named group for that agent that blocks. The order of the groups does not matter, but within a group, the most specific path rule wins when two rules match the same URL.

Blocking AI crawlers: what to consider

AI training crawlers have become a common consideration for publishers over the last few years. Named agents such as GPTBot, ClaudeBot, Bytespider, CCBot, and PerplexityBot fetch pages to build training datasets, retrieval indexes, or both. Some publishers block all of them, some allow them, and some negotiate licensing terms with individual companies. There is no universal right answer, and the choice depends on the content, the audience, and the long-term plans for the site.

The practical options are worth understanding before making the decision. Blocking all AI crawlers removes any chance of the content appearing inside an AI assistant's response, which may reduce referral traffic from those surfaces. Allowing them keeps the content available to a growing set of discovery channels. A middle path is to allow search-oriented AI crawlers that link back to the source and block training-only crawlers that do not. Google-Extended is a useful token in this context: it lets a site opt out of Google's AI training while still allowing Googlebot to crawl and index the site for Search.

The generated file uses standard user-agent groups, so the rules work the same way for AI crawlers as they do for search engines. A site that blocks GPTBot writes a group for GPTBot with a Disallow of the full path. A site that allows GPTBot but blocks Bytespider writes one group for each. Nothing about the AI crawler groups is special from the robots.txt parser's point of view, and the file remains easy to audit and update over time.

Managing crawl budget on a large site

Crawl budget is the informal name for the number of URLs a search engine is willing to fetch from a site in a given window. On a small site it is rarely a problem, because the search engine can fetch every page that matters. On a large site with faceted navigation, internal search pages, session parameters, and duplicate URL patterns, crawl budget becomes a real constraint: the crawler may spend its time on low-value URLs and miss the pages that deserve attention.

Robots.txt is one of several tools for managing crawl budget. Blocking internal search result pages, session identifiers, and calendar or filter combinations keeps the crawler focused on the pages that should rank. Combining that with a clean internal link structure, canonical tags on duplicate URLs, and a sitemap that lists only canonical pages gives the crawler a clear picture of what the site contains.

Crawl-delay is a related but separate directive. It tells a crawler to wait a fixed number of seconds between requests. Bing and Yandex honour it; Google has stated publicly that it does not. On most sites, a crawl delay is unnecessary, and adding one can slow down indexing without any benefit. It is worth considering only when the server is genuinely struggling under crawler load and the hosting cannot be upgraded.

Directives reference

The robots.txt format uses a small set of directives. The generator on this page writes them, and the reference below explains what each one does.

An empty Disallow line, written as Disallow on its own, means everything is allowed for that group. A Disallow of a single slash, written as Disallow: /, means everything is blocked for that group. Both forms are common and both are correct. The Allow directive is matched by the same prefix rules as Disallow, and the more specific rule wins when two rules match the same URL.

A worked example

Consider a publisher in Accra or Nairobi who runs a small editorial site on shared hosting. The site has a public blog, an admin area at /admin/, an internal search at /search/, and a sitemap at the root. The publisher wants Google and Bing to crawl the public blog, wants to keep the admin and search pages out of the crawl, and wants to block GPTBot, ClaudeBot, and CCBot from training on the content.

The resulting file has four named groups and one wildcard group. The named groups block GPTBot, ClaudeBot, and CCBot with Disallow of the full path. The wildcard group allows everything and adds two disallow paths for /admin/ and /search/, plus a sitemap line at the bottom. The file is short, readable, and easy to audit.

A second example is a small business site that wants a firm block on all AI training crawlers. The publisher keeps Googlebot and Bingbot allowed, blocks the same three AI agents, and adds Bytespider and PerplexityBot to the block list. Nothing changes for search engines, and the AI-side policy is explicit. The file remains under twenty lines and can be reviewed in a single read.

A third example is a site that has accidentally been caught by a filter parameter that generates thousands of duplicate URLs. The publisher adds /filter/ to the wildcard disallow, regenerates the file, and re-uploads it. Within a few days, the crawler stops fetching the filtered URLs and reallocates that budget to the canonical pages. No new content was added, no page was edited, and the crawl efficiency improved.

Common mistakes

Blocking CSS and JavaScript. Modern crawlers render pages as a user does, and they need access to CSS, JavaScript, and image assets to understand layout and visible content. Blocking those files leads to a misjudged page. Allow them, and block only paths that genuinely should stay out of the crawl.

Using robots.txt to remove a page from the index. Blocking a URL with robots.txt prevents the crawler from seeing any noindex directive on the page, which can leave the URL in the index without a description. The right sequence is to keep the page crawlable, add a noindex, wait for the search engine to drop the URL, and only then consider blocking it.

Placing the file in the wrong location. Robots.txt only works at the root of the host. A file inside a subfolder does not control the whole domain, and a file on a subdomain does not control the main site. Each host needs its own file.

Using Crawl-delay on Google. Google ignores the directive. If the goal is to slow Google's crawl rate, use the crawl rate settings in Google Search Console. Crawl-delay is useful only for Bing and Yandex.

Blocking the whole site by accident. A Disallow of a single slash blocks everything. If the site needs to be visible in search, that is almost never the intent. Double-check that the wildcard group's Disallow is empty if you intend to allow.

Forgetting the sitemap. The sitemap line is optional, but it is one of the cheapest discovery signals available. Even if the sitemap is submitted through the webmaster tools of each engine, adding it to robots.txt is harmless and sometimes helps crawlers find new URLs faster.

Never reviewing the file again. A robots.txt file is a living document. Folder structures change, privacy policies shift, and new crawlers appear. Review the file whenever the site structure changes, and audit it once a year even if nothing obvious has changed.

Where this tool stops

This generator produces a clean robots.txt file from the options selected on the page. It does not verify that the file is reachable, that the paths listed are correct, that the sitemap URL returns a valid sitemap, or that the chosen rules match the site's actual structure. It does not know how Googlebot or Bingbot will interpret overlapping rules, and it does not model the interaction between robots.txt, canonical tags, noindex directives, or HTTP headers. Those checks belong to the person uploading the file and to the webmaster tools of each search engine.

It also does not provide legal advice on AI training, copyright, or content licensing. Decisions about blocking AI crawlers depend on the publisher's goals, the applicable law in their jurisdiction, and any agreements already in place. The generator gives a way to express those decisions in a standard format, but the decisions themselves belong to the site owner.

What the tool does give you is a fast, transparent, private way to build a root-ready robots.txt file, verify the directives on screen before uploading, and copy or download the result in one step. Used together with the webmaster tools of the search engines that matter, it is a practical starting point for anyone who wants clear, auditable crawl rules.