Definition
An AI crawler is an automated program an AI company runs to fetch web pages. Different crawlers serve different purposes: some gather pages for search features, some fetch a page when a user asks about it, some collect training data. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are examples.
A crawler visit means a page could be used, not that it will be cited. Blocking the wrong crawler, though, can remove a brand from an entire platform’s answers without anyone noticing.
Why it matters
If a platform’s crawler cannot read your pages, nothing else in GEO matters on that platform. It is the first check in any audit.
How it works
Check robots.txt for rules naming each AI crawler, read server logs for their visits and the pages they fetch, and decide deliberately which crawlers to allow for search and which, if any, to block for training.
