The agent directory
98 named agents, crawlers, and log-invisible entities — the same registry that powers Botfall's live classifier. Every record carries provenance: what establishes it, and when it was last checked. No speculative entries, ever.
AI · training crawl 18
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| GPTBot | OpenAI | gptbot | 1 | 11 hours ago |
| ClaudeBot | Anthropic | claudebot | 4 | 8 hours ago |
| anthropic-ai | Anthropic | anthropic-ai | — | not yet seen |
| Claude-Web | Anthropic | claude-web | — | not yet seen |
| CCBot | Common Crawl | ccbot | — | 2 days ago |
| Bytespider | ByteDance | bytespider | 47 | 40 minutes ago |
| meta-externalagent | Meta | meta-externalagent | 59 | 26 minutes ago |
| Amazonbot | Amazon | amazonbot | 36 | 1 hour ago |
| cohere-training-data-crawler | Cohere | cohere-training-data-crawler | — | not yet seen |
| cohere-ai | Cohere | cohere-ai | — | 2 days ago |
| Diffbot | Diffbot | diffbot | — | not yet seen |
| TimpiBot | Timpi | timpibot | — | not yet seen |
| omgilibot | Webz.io | omgilibot | — | not yet seen |
| ImagesiftBot | Hive | imagesiftbot | — | not yet seen |
| GoogleOther | googleother | — | 3 days ago | |
| AI2Bot | AI2 | ai2bot | — | not yet seen |
| Google-Extended | log-invisible (robots token only) | — | not yet seen | |
| Applebot-Extended | Apple | log-invisible (robots token only) | — | not yet seen |
Search indexing 28
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | oai-searchbot | 1 | 20 hours ago |
| Claude-SearchBot | Anthropic | claude-searchbot | — | 2 days ago |
| PerplexityBot | Perplexity | perplexitybot | — | 2 days ago |
| DuckAssistBot | DuckDuckGo | duckassistbot | — | 2 days ago |
| Google-CloudVertexBot | google-cloudvertexbot | — | 23 days ago | |
| YouBot | You.com | youbot | — | 2 days ago |
| PhindBot | Phind | phindbot | — | not yet seen |
| Meta-WebIndexer | Meta | meta-webindexer | 1 | 13 hours ago |
| Amzn-SearchBot | Amazon | amzn-searchbot | — | 20 days ago |
| Storebot-Google | storebot-google | — | 27 days ago | |
| Googlebot-Image | googlebot-image | — | not yet seen | |
| Googlebot-Video | googlebot-video | — | not yet seen | |
| Googlebot | googlebot | 8 | 1 hour ago | |
| FeedFetcher-Google | feedfetcher-google | — | 48 days ago | |
| GoogleProducer | googleproducer | — | not yet seen | |
| BingVideoPreview | Microsoft | bingvideopreview | — | not yet seen |
| Bingbot | Microsoft | bingbot | 6 | 32 minutes ago |
| msnbot | Microsoft | msnbot | — | not yet seen |
| Applebot | Apple | applebot | 1 | 7 hours ago |
| DuckDuckBot | DuckDuckGo | duckduckbot | 1 | 10 hours ago |
| YandexBot | Yandex | yandex | — | 23 days ago |
| Baiduspider | Baidu | baiduspider | 6 | 6 hours ago |
| PetalBot | Huawei | petalbot | 52 | 4 hours ago |
| Slurp | Yahoo | slurp | — | not yet seen |
| SeznamBot | Seznam | seznambot | — | not yet seen |
| Sogou web spider | Sogou | sogou | 4 | 1 hour ago |
| IbouBot | Ibou (Babbar SAS, France) | iboubot | — | 1 day ago |
| MojeekBot | Mojeek Ltd (Mojeek search engine, UK) | mojeekbot | — | 6 days ago |
AI · live answer fetch 8
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | chatgpt-user | 8 | 4 hours ago |
| Claude-User | Anthropic | claude-user | — | 2 days ago |
| Perplexity-User | Perplexity | perplexity-user | — | 2 days ago |
| MistralAI-User | Mistral | mistralai-user | — | 2 days ago |
| Meta-ExternalFetcher | Meta | meta-externalfetcher | — | not yet seen |
| Amzn-User | Amazon | amzn-user | — | not yet seen |
| Google-GeminiNotebook | google-gemininotebook | — | not yet seen | |
| ShapBot | Parallel Web Systems (parallel.ai) | shapbot | — | 10 days ago |
AI · agentic browser 3
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| Google-Agent | google-agent | — | not yet seen | |
| HeadlessChrome | unknown | headlesschrome | 1 | 2 hours ago |
| PhantomJS | unknown | phantomjs | — | not yet seen |
Social link preview 10
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| facebookexternalhit | Meta | facebookexternalhit | 1 | 22 hours ago |
| Twitterbot | X | twitterbot | 1 | 20 hours ago |
| LinkedInBot | linkedinbot | — | 20 days ago | |
| Pinterestbot | pinterestbot | — | 6 days ago | |
| Slackbot | Slack | slackbot | — | 20 days ago |
| Discordbot | Discord | discordbot | — | 20 days ago |
| Meta | whatsapp | — | 20 days ago | |
| TelegramBot | Telegram | telegrambot | — | 20 days ago |
| GoogleMessages | googlemessages | — | not yet seen | |
| MicrosoftPreview | Microsoft | microsoftpreview | — | not yet seen |
Ad-quality crawl 5
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| AdsBot-Google-Mobile | adsbot-google-mobile | — | 14 days ago | |
| AdsBot-Google | adsbot-google | — | not yet seen | |
| Mediapartners-Google | mediapartners-google | — | not yet seen | |
| AdIdxBot | Microsoft | adidxbot | — | not yet seen |
| Meta-ExternalAds | Meta | meta-externalads | 1 | 22 hours ago |
Site diagnostics 2
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| Google-InspectionTool | google-inspectiontool | — | 53 days ago | |
| Google-Site-Verification | google-site-verification | — | not yet seen |
SEO tooling 7
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| SemrushBot | Semrush | semrushbot | 2 | 17 hours ago |
| AhrefsBot | Ahrefs | ahrefsbot | 2 | 15 hours ago |
| DotBot | Moz | dotbot | 1 | 5 hours ago |
| MJ12bot | Majestic | mj12bot | — | 2 days ago |
| DataForSeoBot | DataForSEO | dataforseobot | — | 7 days ago |
| SERankingBacklinksBot | SE Ranking | seranking | — | 5 days ago |
| WebatlaBot | webatla (webatla.com) | webatlabot | — | 29 days ago |
Monitoring 5
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| UptimeRobot | monitoring | uptimerobot | — | not yet seen |
| Pingdom | monitoring | pingdom | — | not yet seen |
| StatusCake | monitoring | statuscake | — | not yet seen |
| Datadog | monitoring | datadog | — | not yet seen |
| AwarioBot (AwarioSmartBot / AwarioRssBot family) | Awario (social listening / brand monitoring platform) | awariobot | — | 2 days ago |
Web archive 2
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| ia_archiver | Internet Archive | ia_archiver | — | not yet seen |
| archive.org_bot | Internet Archive | archive.org_bot | — | not yet seen |
Exploit scanner 2
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| RecordedFuture Global Inventory Crawler | Recorded Future, Inc. (Mastercard subsidiary) | recordedfuture global inventory crawler | — | 7 days ago |
| BitSightBot | Bitsight Technologies, Inc. | bitsightbot | 1 | 2 hours ago |
Other bots 8
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| OAI-AdsBot | OpenAI | oai-adsbot | — | not yet seen |
| APIs-Google | apis-google | — | not yet seen | |
| Google-Safety | google-safety | — | not yet seen | |
| Google-Read-Aloud | google-read-aloud | — | 5 days ago | |
| Google-CWS | google-cws | — | not yet seen | |
| Google-Pinpoint | google-pinpoint | — | not yet seen | |
| TikTokSpider | TikTok / ByteDance | tiktokspider | — | 3 days ago |
| ZoominfoBot | ZoomInfo (Zoom Information Inc. / ZoomInfo Technologies) | zoominfobot | 1 | 19 hours ago |