Does blocking a page in robots.txt keep it out of Google?

No. Disallow stops crawling, not indexing: a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out, let Google crawl it and add a noindex robots meta tag or X-Robots-Tag header.

Why do my rules for User-agent: * not apply to Googlebot?

A crawler follows only the most specific group that names it. If your file has a User-agent: Googlebot group, Googlebot ignores the * group entirely, so repeat the shared rules in its group. The tool warns when this happens.

How do I stop AI companies training on my content but stay in AI search?

Block the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) and allow the search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot). The Crawlers tab shows where each one stands on your site.

How long until Google sees a robots.txt change?

Google usually caches robots.txt for up to a day. You can ask for a refresh in Search Console’s robots.txt report.

Is the site I check stored or logged?

No. Files are fetched through a small proxy that keeps no logs or copies, and pasted files never leave your browser.

Robots.txt Tester & Sitemap Checker

User guide

What this tool does

The Robots/Sitemap Inspector reads a site’s robots.txt the way Google does. It shows every line with its problems marked, tells you whether any URL is allowed or blocked for any crawler and which line decides it, and shows at a glance where search engines and AI crawlers stand. It then checks every XML sitemap the file lists: broken XML, size limits, bad dates, URLs on the wrong host and, most usefully, pages that the sitemap asks Google to crawl but robots.txt blocks.

When to use it

When pages aren’t being crawled or indexed: find the rule that blocks them.
Before changing robots.txt: paste your draft and test the important URLs first. One wrong Disallow: / can take a site out of search.
When deciding on AI crawlers: see which AI training and AI search crawlers can reach your content.
After a migration or CMS change: sitemaps often keep old hosts, http:// URLs or blocked sections.

How to use it

Choose the input.
- Live site: type the site’s address and press Check site. The tool reads /robots.txt and the first sitemap it lists, or /sitemap.xml if none is listed. If you give a page’s address, that page is tested straight away.
- Paste files: paste a robots.txt, a sitemap, or both. Useful for drafts, staging sites and gzip-compressed sitemaps (paste the uncompressed XML). Add the site address so the sitemap checks can spot URLs on the wrong host.
Test a URL. Enter a path or full URL, pick a crawler, and read the verdict. The deciding line is highlighted in the file below.
Read the file. Lines with problems are marked red (error), amber (warning) or blue (note), with the explanation underneath.
Check the Crawlers tab to see which search engines and AI crawlers may fetch a path, and which line or group decides it.
Check the Sitemaps tab. Click a sitemap to check it. For a sitemap index, its child sitemaps appear in the list: click any of them to check it.

How crawlers read robots.txt

One group per crawler. A crawler picks the group whose User-agent names it most specifically, and ignores all the others, including User-agent: *. Rules you want everywhere must be repeated in each named group.
Google’s crawlers fall back to Googlebot. Googlebot-Image, for example, uses a Googlebot group when there’s no group of its own.
The longest matching rule wins. Allow: /shop/sale/ beats Disallow: /shop/. When an Allow and a Disallow are the same length, Allow wins.
Wildcards: * matches any characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF.
Paths are case-sensitive. Disallow: /Admin/ doesn’t block /admin/.
Missing file: if robots.txt returns 404, everything may be crawled. If it returns a 5xx server error, Google stops crawling the site until it’s fixed.
Google ignores Crawl-delay, Host, and Noindex lines in robots.txt.

AI crawlers

Most AI companies run separate crawlers for training models, for building their AI search index, and for fetching a page a user asked about. Blocking one doesn’t block the others.

Purpose	Crawlers
Model training	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider
AI search	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot
On a user’s request	ChatGPT-User, Claude-User, Perplexity-User

A common choice is to block training and allow AI search:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Blocking Google-Extended does not affect Google Search; that is controlled by Googlebot.

What the sitemap checks cover

Check	Why it matters
Valid XML with a `<urlset>` or `<sitemapindex>` root	A broken file is rejected whole
At most 50,000 URLs and 50 MB per file	Larger files are cut off; split them and use a sitemap index
Full, absolute URLs on the sitemap’s own host and protocol	Other hosts need separate verification; `http://` entries on an `https://` site send mixed signals
No duplicates or `#fragments`	Wasted crawl and diluted signals
`<lastmod>` in W3C format, not in the future, not all identical	Google only uses lastmod when it’s accurate
URLs not blocked by robots.txt	Google can’t crawl them, so listing them is a contradiction

<changefreq> and <priority> are reported but not counted as problems: Google ignores them.

Limits

The checker reads up to 5 MB of a file. Gzip-compressed sitemaps (.xml.gz) can’t be fetched: paste the XML instead.
It doesn’t fetch the URLs in a sitemap, so it can’t tell if they return errors or redirects. The Hreflang/Canonical Validator checks individual pages.
Crawlers that don’t follow the standard can read robots.txt differently. Robots.txt is a request, not access control: use authentication for anything private.

FAQ

Does blocking a page in robots.txt keep it out of Google?: No. Disallow stops crawling, not indexing: a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out, let Google crawl it and add a noindex robots meta tag or X-Robots-Tag header.
Why do my rules for User-agent: * not apply to Googlebot?: A crawler follows only the most specific group that names it. If your file has a User-agent: Googlebot group, Googlebot ignores the * group entirely, so repeat the shared rules in its group. The tool warns when this happens.
How do I stop AI companies training on my content but stay in AI search?: Block the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) and allow the search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot). The Crawlers tab shows where each one stands on your site.
How long until Google sees a robots.txt change?: Google usually caches robots.txt for up to a day. You can ask for a refresh in Search Console’s robots.txt report.
Is the site I check stored or logged?: No. Files are fetched through a small proxy that keeps no logs or copies, and pasted files never leave your browser.