Robots.txt Generator and Tester
Assemble a robots.txt from user-agent groups and rules, then paste a URL into the tester to see exactly which line decides it. The tester implements the RFC 9309 precedence rules — longest matching path wins, Allow wins ties, * and $ behave as the spec defines — and shows every rule that matched, not just the verdict.
Build your robots.txt
robots.txt
User-agent: * Disallow:
Upload this to the root of your domain — https://example.com/robots.txt. A file in a subdirectory is not read.
Test a URL against these rules
Allowed — matched the “User-agent: *” group
No rule in the “*” group matches /, so it is allowed by default.
The tester implements RFC 9309: longest matching path wins, Allow wins ties, * is a wildcard and $ anchors the end of the path. It models the documented spec, not any one crawler’s internal behavior — and robots.txt controls crawling, not indexing, so a blocked URL can still appear in search results if other pages link to it.
Design the pages you are letting crawlers have
Moda builds and hosts fast, crawlable marketing sites — landing pages, docs, and campaign microsites.
Try Moda free →The precedence rules people get wrong
Rules are not evaluated top to bottom. Every rule in the applicable group is matched against the URL path, and the most specific match wins — where specificity is simply the character length of the rule’s path pattern. When an Allow and a Disallow tie on length, Allow wins. That is why “Disallow: /assets/” plus “Allow: /assets/public/” lets the public subfolder through: the Allow pattern is longer. Ordering the lines differently changes nothing, which surprises people who expect firewall-style first-match semantics.
Wildcards, anchors, and the empty Disallow
An asterisk matches any run of characters, so “Disallow: /*?” blocks every URL with a query string. A trailing $ anchors the end of the path, so “Disallow: /*.pdf$” blocks /report.pdf but not /report.pdf?download=1. Everything else is literal — a rule is a prefix match against the path and query, never a regular expression. And “Disallow:” with no value is not a rule at all: it is the spec’s way of saying nothing is disallowed, which is why this tool emits it rather than dropping it.
Groups apply to one crawler, not all of them
A crawler reads exactly one group: the one whose user-agent token matches its own name, falling back to “User-agent: *” only when no named group matches. It does not merge the catch-all group with its own. So if you add a Googlebot group with a single rule, Googlebot stops obeying everything in your * group — a common way to accidentally unblock an entire admin area. Groups that repeat the same user-agent token are merged together, which the tester shows you.
What robots.txt cannot do
It controls crawling, not indexing. A URL you disallow can still appear in search results — without a snippet — when other pages link to it, because the crawler never fetches the page to see a noindex tag. Keeping a page out of the index means letting it be crawled and serving “noindex”, or putting it behind authentication. robots.txt is also public and advisory: it tells well-behaved crawlers what to skip and does nothing about the ones that ignore it, so it is never a security control.
Frequently asked questions
Where does robots.txt have to live?
At the root of each host and scheme: https://example.com/robots.txt covers https://example.com, but not http://example.com, not www.example.com if that is a separate host, and not a subdomain. A file at /blog/robots.txt is never read.
Does this tester match what Googlebot actually does?
It implements RFC 9309, the published Robots Exclusion Protocol — the same precedence, wildcard, and anchoring rules Google documents. It is a model of the specification, not a copy of any crawler’s internal code, so treat a surprising result as a reason to check your file against the live tester in Search Console before shipping.
Is Crawl-delay honored?
It is not part of RFC 9309. Bing and Yandex read it; Google ignores it entirely and asks you to set a crawl rate in Search Console instead. The generator emits it because it costs nothing for the crawlers that do use it, and the tester parses it without pretending it affects the allow/block verdict.
How do I block AI crawlers?
They are ordinary user-agent tokens: add a group for GPTBot, CCBot, ChatGPT-User, or whichever you mean, with “Disallow: /”. The preset buttons add the common ones. Remember that each named group replaces the catch-all for that crawler, so a named group needs every rule you want that crawler to obey.
What happens if my robots.txt has a syntax error?
Crawlers skip lines they cannot parse and use the rest, which is why a typo like “Dissallow” silently exposes a directory. The tester lists every line it could not use, with the line number, so a broken directive shows up as a warning instead of a mystery.