AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended): allow or block?
AI assistants read the web using named crawlers, and your robots.txt decides which ones you let in. For a local business trying to appear in AI answers, this is a simple but consequential setting. AI answers change daily; a scan is a snapshot, but crawler access is a one-time fix with lasting effect.
Who the main AI crawlers are — and which ones decide whether you can be found
- OAI-SearchBot — OpenAI's crawler that surfaces websites in ChatGPT's search features. This is the one that matters for appearing in ChatGPT answers.
- GPTBot — OpenAI's training crawler. OpenAI says disallowing GPTBot does not affect whether you appear in ChatGPT search results.
- ChatGPT-User — fetches a page when a ChatGPT user asks about it; not an automatic crawl.
- Claude-SearchBot / ClaudeBot / Claude-User — Anthropic's search crawler, training crawler, and on-demand fetcher, respectively. All three honor robots.txt.
- PerplexityBot — Perplexity's crawler for its answer index.
- Google-Extended — Google's control for Gemini training and grounding in Google's other systems. Google states it does not affect a site's inclusion or ranking in Google Search. AI Overviews and AI Mode are Search features that follow Googlebot and your snippet controls instead.
- CCBot — Common Crawl, an open dataset many AI systems train on.
- Applebot-Extended — Apple's control for its AI use.
The decision, for a local service business
If your goal is discovery — being found and named — allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking them opts you out of being read by those assistants, which is the opposite of what a contractor wants. The training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) are a separate choice about whether your pages may help train models; some publishers block them to protect original content. A local contractor's pages are marketing, not a paywalled archive, so most contractors simply allow everything.
How to allow them
In your robots.txt, make sure there's no blanket block on these agents. To be explicit, you can add:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Then confirm no earlier Disallow: / rule for these agents overrides it.
The honest limit
Allowing crawlers makes you eligible to be read — it does not guarantee you'll be named. It's a necessary baseline, not a lever that forces an outcome.
FAQ
- Will allowing AI crawlers hurt my Google ranking?
- No. These bots are separate from Googlebot, and Google says Google-Extended is not used as a ranking signal.
- If I block them, am I invisible to ChatGPT?
- Blocking OAI-SearchBot removes you from ChatGPT's search results; blocking only GPTBot affects model training, not search appearance, according to OpenAI.
- Does allowing crawlers mean AI will recommend me?
- No — it makes you eligible to be read. AI answers change daily; a scan is a snapshot, and access alone guarantees nothing.
- How do I know if my site currently blocks them?
- Check your robots.txt for Disallow rules on these agents. A scan can flag AI-crawler access for you.