AI companies now run two kinds of crawler: search crawlers, which decide whether your pages can appear and be linked in AI answers, and training crawlers, which collect content to build future AI models. If you want customers to find you through ChatGPT, Claude or Perplexity, allow their search crawlers. Whether you allow training crawlers is a separate business decision, and blocking them doesn’t affect your visibility in search.
Getting this wrong is surprisingly common. We regularly find small business websites that block every AI bot because of a well-meant firewall setting, and then wonder why ChatGPT never mentions them.
The short version
- Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you can appear in those tools’ search answers. Usually, allow them.
- Training crawlers such as GPTBot and ClaudeBot collect content for model training. Blocking them doesn’t remove you from search answers.
- Google-Extended isn’t a separate crawler. It’s a setting that limits Google’s use of your content for Gemini training and grounding in some other Google systems. It doesn’t affect Google Search or AI Overviews.
- Check your firewall and CDN as well as robots.txt. In July 2025, Cloudflare started blocking AI crawlers by default.
In this article
Search crawlers and training crawlers
When an AI assistant answers a question using the web, it relies on an index built by its search crawler, or fetches pages live when a user asks. That’s how your business ends up cited with a link. Training crawlers are different. They gather text that may be used to train the next version of a model, and they don’t send visitors.
The big AI companies now separate these jobs into different user agents, so you can choose for each one.
The main crawlers at a glance
| Crawler | Company | What it does | Block it and… |
|---|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google. Don’t block it | |
| Google-Extended | A setting, not a crawler: limits use for Gemini training and grounding in some other Google systems | Google Search and AI Overviews are unaffected | |
| Bingbot | Microsoft | Bing search and Microsoft Copilot | You disappear from Bing and Copilot |
| OAI-SearchBot | OpenAI | ChatGPT search | You won’t be shown in ChatGPT search answers |
| GPTBot | OpenAI | Content that may be used to train OpenAI’s models | Training stops. Search visibility unaffected |
| ChatGPT-User | OpenAI | Visits pages when a user asks ChatGPT to | robots.txt may not apply, as it’s user-initiated |
| Claude-SearchBot | Anthropic | Search results in Claude | Less visibility in Claude’s search answers |
| ClaudeBot | Anthropic | Content that may be used to train Anthropic’s models | Training stops |
| Claude-User | Anthropic | Visits pages when a user asks Claude to | Less visibility in user-directed searches |
| PerplexityBot | Perplexity | Perplexity’s search index | Less visibility in Perplexity’s answers |
| CCBot | Common Crawl | An open web archive used by many AI developers for training | Your content stays out of future Common Crawl archives |
OpenAI says its settings are “independent of the others”, so you can allow OAI-SearchBot and block GPTBot. It also says changes to robots.txt take about 24 hours to reach its systems.
A robots.txt you can adapt
This example allows all search crawlers but asks the main training crawlers to stay away. Change it to suit your own choices.
# Training crawlers: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else, including search crawlers: welcome
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.co.uk/sitemap.xmlA few things to know:
- robots.txt is a request, not a lock. The major AI companies say they follow it, but badly behaved bots ignore it. Use your firewall for those.
- Each subdomain needs its own file. shop.example.co.uk doesn’t follow the rules on www.example.co.uk.
- Be specific. A rule that names a crawler directly overrides the general
*rules for that crawler. - Test it. Search Console’s robots.txt report shows how Google reads your file.
Don’t forget your firewall and CDN
Lots of websites are protected by services such as Cloudflare, which sit in front of your site and can block bots before they ever see robots.txt. In July 2025, Cloudflare announced it would block AI crawlers by default and introduced “pay per crawl”, which lets site owners charge AI companies for access.
That’s useful for publishers who want to protect their content. For a business that wants to be recommended, it can quietly make you invisible to AI search. If your site uses Cloudflare or a similar service:
- Check its bot or AI crawler settings.
- Allow the search crawlers you want, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Decide separately about training crawlers.
- Check your server logs, or ask your host, to confirm the crawlers you allowed are actually getting through.
How to decide
Most small businesses that sell services or products should allow every search crawler. Being cited and linked in AI answers brings in customers, and there’s little downside.
Training crawlers are a judgement call. Some businesses are happy for their public marketing pages to help train AI models, as it may help those models know more about them. Others would rather their content wasn’t used without payment. Blocking training crawlers protects future use of your content but won’t remove anything already collected.
Publishers and creators whose content is the product, such as journalists, course creators and photographers, have stronger reasons to block training and consider paid access.
Whatever you choose, write it down, apply it consistently and review it every six months. New crawlers appear regularly.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI’s training crawler. Whether you appear in ChatGPT’s search answers depends on OAI-SearchBot, which you can allow separately.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode are part of Google Search and use Googlebot. Google-Extended only limits use of your content for Gemini training and grounding in some of Google’s other systems.
How do I know which AI bots visit my site?
Your server logs record every visit along with the user agent, which names the crawler. Your hosting company can usually show you a report. Some CDNs, including Cloudflare, also show AI crawler activity in their dashboards.
Is blocking AI crawlers legal?
Yes. It’s your website, and robots.txt is the standard way to tell crawlers what they may access. The major AI companies say they respect it.
Ready for some help?
Not sure what your robots.txt or firewall is telling AI crawlers? Book a free chat and we’ll check which AI search tools can actually see your site. It’s one of the first checks in our GEO service.
References
- OpenAI, Overview of OpenAI crawlers.
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central, AI features and your website, last updated December 2025.
- MIT Technology Review, Cloudflare will now, by default, block AI bots from crawling its clients’ websites, 1 July 2025.
- Google Search Central, Introduction to robots.txt.