Which AI crawlers should a health practice let in?

What is robots.txt, and what can it actually do?

robots.txt is a plain text file at the top of your website, at yourdomain.com/robots.txt, that tells automated crawlers which parts of the site they may visit. The rules for it are an internet standard, RFC 9309, published by the IETF in 2022. The standard says the file must sit in the top-level path of the site and be named robots.txt in lowercase.¹

Two parts of that standard matter for a practice owner:

• The rules are not security. RFC 9309 states that they are not a form of access authorization.¹ A well-behaved crawler follows them. Nothing forces a badly behaved one to.
• Changes are not instant. Crawlers should not rely on a cached copy of your file for more than 24 hours, so a fix usually takes a day to be picked up.¹ OpenAI gives the same figure for its own systems: about 24 hours after you update the file.²

Each block in the file starts with a User-agent line naming a crawler, followed by Allow or Disallow lines listing paths. A line reading Disallow with a single slash under a crawler's name blocks that crawler from the whole site.

Which AI crawlers are there, and what does each one do?

Every major AI company now runs more than one crawler, and they do different jobs. The names below are taken from each operator's own documentation, checked in September 2026.

Sources for each row: Google,⁴ OpenAI,² Anthropic³ and Perplexity.⁵ Perplexity states that PerplexityBot is not used to crawl content for AI foundation models.⁵

Why do the search crawlers matter more than they used to?

Because more patients now read an answer instead of choosing a link. Pew Research Center tracked the real browsing of 900 U.S. adults in March 2025. About 18 percent of their Google searches produced an AI summary.⁶ When one appeared, users clicked a traditional result in 8 percent of visits, against 15 percent when it did not. They clicked a link inside the summary itself in just 1 percent of visits.⁶

Read that from a practice's side. When an answer appears, being named inside it is often the whole visit. A crawler that cannot read your site cannot put your hours, your services or your credentials into that answer, and the patient rarely scrolls past it to find you.

What happens if you block each type?

The operators document the consequences themselves:

• OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.²
• Anthropic says blocking Claude-SearchBot prevents your content being indexed for search, and blocking Claude-User prevents retrieval when a user asks Claude a question.³
• OpenAI says disallowing GPTBot indicates your content should not be used to train its foundation models.² Anthropic says blocking ClaudeBot excludes future material from training datasets.³
• Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.⁴

What robots.txt cannot do

The fetchers that open a page because a user asked are treated differently. OpenAI writes that for ChatGPT-User, because the action is initiated by a user, robots.txt rules may not apply.² Perplexity states that Perplexity-User generally ignores robots.txt rules.⁵

So robots.txt is the wrong tool for keeping anything private. Intake forms, patient portals, pricing sheets meant for referral partners and staff documents belong behind a login. If a page must not be read by anyone outside the practice, it should not be public in the first place.

Should a practice allow the training crawlers?

For most practices, yes. Your service pages, your practitioner bios and your articles are public because you want people to understand what you do. Having that understanding built into future models costs you little. Blocking training does not remove you from search answers either way, per the documentation above.

There are reasonable exceptions:

• A practice that sells a paid course, a protocol library or membership content that happens to sit on public pages. Move it behind a login first; blocking crawlers is the weaker fix.
• A practitioner who objects in principle to model training on their writing. That is a legitimate position, and blocking GPTBot, ClaudeBot and Google-Extended expresses it without costing search visibility.

What is not reasonable is blocking everything by accident. A single rule under User-agent with an asterisk and Disallow with a slash tells every compliant crawler, search engines included, to stay out of the whole site.

How do you check your own robots.txt in ten minutes?

1. Open yourdomain.com/robots.txt in a browser. If it returns a 404, no crawler is restricted by it, which is fine for most small sites.
2. Look for a block that names every crawler with an asterisk and disallows the root. If you see one, fix it first.
3. Search the file for each search crawler in the table above. Any of them followed by a full Disallow means you are absent from that tool's answers.
4. Decide on the training crawlers deliberately, and write the decision down so the next web developer does not undo it.
5. Ask whoever manages your hosting or security settings whether bots are blocked before the file is even read. Some firewall settings do this.
6. After any change, allow about a day before testing again.²

Crawler names change, and new ones appear. Put a yearly reminder in the calendar to compare your file against each operator's documentation. This check is one of the steps in our free check of your practice, and the wider order of work is in the pillar guide on becoming the practice AI recommends. Once crawlers can read the site, structured data is how you tell them what they are reading.