Robots.txt Tester

Check that your robots.txt is valid and is not blocking your site. Then test any page against it.

or to test it before it goes live

It's free to run

What this robots.txt tester checks

It starts with two questions. Is the file valid? And is it shutting crawlers out of your site?

For the first one it reads every line. A misspelled directive, a rule with no group, a path that can never match, a line crawlers skip: each one is listed with its line number. That's the robots.txt validator part.

For the second it applies the rules the way a crawler does. If one line closes the whole site, you see that at the top, with the line that does it. A table then shows every search and AI crawler and whether it can get in.

You can also test a single page. Pick the URL and the crawler. The tester shows Allowed or Blocked and highlights the rule that decided it.

Once the file is fetched, all the checking happens in your browser. A file you paste or edit never leaves the page.

How to test a robots.txt file

  1. Enter your site or a link to its robots.txt. The tester fetches the live file from that host.
  2. Read the two lines at the top. One tells you if the file is valid. The other tells you if your site is open to crawlers or blocked.
  3. Fix what is listed under Validation. Every problem has a line number. Click it to go to that line.
  4. Check the crawler table. It shows which search and AI crawlers can access your site, and the rule behind each answer.
  5. Test a specific page. Press the button, enter the page and pick a crawler such as Googlebot or GPTBot. You get Allowed or Blocked and the line that decided it.
  6. Edit and test again. Change a rule and the results update as you type. Once the file says what you want, copy it to your site.

You can skip the first step. Paste a robots.txt draft to check it before it is published. That's safer than changing a live file every crawler reads.

How robots.txt rules are matched

Most surprises come from a few rules the file itself never states. The tester applies them the way the published standard and Google's documentation describe.

One group per crawler

A crawler looks for a group that names it. If it finds one, it follows that group and ignores User-agent: * entirely. The two are never combined. So if you block /admin/ for everyone and then add a short group for GPTBot without repeating the block, /admin/ is open to GPTBot.

The longest rule wins

When several rules match a URL, the most specific one decides, and specific means longest. The order of the lines doesn't matter. If an Allow and a Disallow of the same length both match, Allow wins.

Wildcards and the end anchor

* stands for any run of characters and $ marks the end of the URL. Every rule matches from the start of the path, so Disallow: /blog also blocks /blog-archive/ and /blog.html.

RuleBlocksDoes not block
Disallow: /Every URL on the hostNothing
Disallow:Nothing. An empty value allows everythingEvery URL
Disallow: /blog/blog, /blog/post, /blog-archive//news/blog
Disallow: /blog//blog/post/blog, /blog-archive/
Disallow: /*.pdf$/files/guide.pdf/files/guide.pdf?v=2
Disallow: /*?Any URL with a query string/shop/

Paths are case-sensitive

Directive names and user agent names can be written in any case. Paths can't. Disallow: /Private/ does nothing for /private/.

One file per host

The file has to sit at the root and be named robots.txt. It only covers the host it is served from, so shop.example.com and www.example.com each need their own.

Test robots.txt for AI crawlers

AI assistants read your site with their own crawlers, and each one looks for its own name in robots.txt. GPTBot collects pages for training. OAI-SearchBot builds the index behind ChatGPT search. Anthropic splits ClaudeBot and Claude-SearchBot the same way. Blocking one is a different decision from blocking another.

A lot of sites carry rules for these crawlers that nobody on the team wrote. A plugin, a hosting default or a template added them. The crawler table shows what your file tells each of them, and the page test checks one crawler on one URL.

The group rule above matters most here. A file that names GPTBot only to keep it out of one folder has also lifted every rule under User-agent: * for it.

robots.txt is also only the first layer. A firewall or CDN can refuse a crawler your file invites in, and nothing in robots.txt will show it. The AI Crawlability Checker sends a real request as each AI crawler and shows what your server answers.

Common robots.txt mistakes

A leftover Disallow: /. Staging sites block everything on purpose. Then the site launches with that rule still in the file, and every page drops out of crawling.

Using robots.txt to hide a page. It stops crawling, not indexing. A blocked URL that other pages link to can still appear in results. A page that has to stay out of search needs a noindex, and crawlers must be able to reach the page to see it.

Blocking CSS and JavaScript. Google renders a page before it judges it. If the files that build the page are disallowed, Google sees a broken version.

Rules above the first User-agent line. A rule has to belong to a group. Anything written before a group starts is ignored.

A path without a leading slash. Disallow: admin matches nothing. It has to be Disallow: /admin.

Noindex inside robots.txt. Google doesn't support it there. It belongs in a meta tag or an X-Robots-Tag header.

Relying on Crawl-delay. Google ignores it. Bing and some other crawlers follow it.

A robots.txt that returns an error. A missing file means crawlers may fetch everything. A server error is a different story: Google treats it as a block on the whole site until the file comes back.

Blocked by robots.txt in Search Console

Two statuses in Google Search Console point at this file, and they call for opposite fixes.

Blocked by robots.txt

Google tried to crawl the URL and a Disallow rule stopped it. If that's what you wanted, there is nothing to fix. If the page should be in search, test it here with Googlebot selected, find the line that blocks it, then remove the rule or narrow it. Once the file is updated, ask Google to recrawl the page.

Indexed, though blocked by robots.txt

Google put the URL in its index without reading it, usually because other pages link to it. The result is a bare listing with no description. What to do next depends on what you want.

If the page should be in search, remove the Disallow so Google can read it. If the page should be gone, do the thing that feels backwards: allow crawling and add a noindex tag. Google can't see a noindex on a page it isn't allowed to fetch, so blocking and noindexing at once keeps the URL in the index.

Frequently asked questions

How do I test my robots.txt file?

Enter your site in the tester above. It fetches the live file and tells you if it is valid and whether it blocks your site. To check one page, press Test a specific page and pick a crawler. To try a change before you publish it, edit the file in the tester or paste a new version.

What does "Blocked by robots.txt" mean in Search Console?

Google tried to crawl the URL and a Disallow rule in your robots.txt told it not to. If that's what you want, nothing is wrong. If you want the page in search, find the rule with the tester, remove or narrow it, and ask Google to recrawl the page.

What does "Indexed, though blocked by robots.txt" mean?

Google added the URL to its index without reading the page, usually because other pages link to it. If you want the page in search, remove the Disallow rule. If you want it gone, allow crawling and add a noindex tag. Google can't see a noindex on a page it isn't allowed to fetch.

Does robots.txt stop a page from being indexed?

No. robots.txt controls crawling, not indexing. A blocked page can still show up in results as a bare link. To keep a page out of the index, let crawlers reach it and mark it noindex.

Does a rule for one crawler add to the rules for all crawlers?

No, it replaces them. A crawler that finds a group with its own name follows only that group and ignores User-agent: * completely. Anything you want that crawler to stay out of has to be repeated in its own group.

Which rule wins when Allow and Disallow both match?

The longest matching rule wins, wherever it sits in the file. If an Allow and a Disallow of the same length both match, Allow wins. The order of the lines doesn't matter.

Do AI crawlers like GPTBot follow robots.txt?

The major operators say their crawlers do, and each one reads the group with its own name. Agents that fetch a page because a person asked are a different case and may not follow the file. robots.txt is a request, so a block that has to hold belongs in the firewall.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers where they may go. llms.txt is a proposed file that lists the pages you'd like AI models to read. It doesn't allow or block anything, and no major assistant has confirmed using it.

A clean robots.txt lets the crawlers in. Whether the assistants then name your brand is a separate question, and the AI Visibility check answers it.

Check your AI visibility