The agent directory
89 named agents, crawlers, and log-invisible entities — the same registry that powers Botfall's live classifier. Every record carries provenance: what establishes it, and when it was last checked. No speculative entries, ever.
AI · training crawl 18
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| GPTBot | OpenAI | gptbot | — | not yet seen |
| ClaudeBot | Anthropic | claudebot | 4 | 1 hour ago |
| anthropic-ai | Anthropic | anthropic-ai | — | not yet seen |
| Claude-Web | Anthropic | claude-web | — | not yet seen |
| CCBot | Common Crawl | ccbot | — | not yet seen |
| Bytespider | ByteDance | bytespider | 17 | 4 hours ago |
| meta-externalagent | Meta | meta-externalagent | 67 | 1 minute ago |
| Amazonbot | Amazon | amazonbot | 13 | 1 hour ago |
| cohere-training-data-crawler | Cohere | cohere-training-data-crawler | — | not yet seen |
| cohere-ai | Cohere | cohere-ai | — | not yet seen |
| Diffbot | Diffbot | diffbot | — | not yet seen |
| TimpiBot | Timpi | timpibot | — | not yet seen |
| omgilibot | Webz.io | omgilibot | — | not yet seen |
| ImagesiftBot | Hive | imagesiftbot | — | not yet seen |
| GoogleOther | googleother | — | not yet seen | |
| AI2Bot | AI2 | ai2bot | — | not yet seen |
| Google-Extended | log-invisible (robots token only) | — | not yet seen | |
| Applebot-Extended | Apple | log-invisible (robots token only) | — | not yet seen |
Search indexing 26
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | oai-searchbot | 1 | 11 hours ago |
| Claude-SearchBot | Anthropic | claude-searchbot | — | not yet seen |
| PerplexityBot | Perplexity | perplexitybot | — | not yet seen |
| DuckAssistBot | DuckDuckGo | duckassistbot | — | not yet seen |
| Google-CloudVertexBot | google-cloudvertexbot | — | not yet seen | |
| YouBot | You.com | youbot | — | not yet seen |
| PhindBot | Phind | phindbot | — | not yet seen |
| Meta-WebIndexer | Meta | meta-webindexer | — | not yet seen |
| Amzn-SearchBot | Amazon | amzn-searchbot | — | not yet seen |
| Storebot-Google | storebot-google | — | not yet seen | |
| Googlebot-Image | googlebot-image | — | not yet seen | |
| Googlebot-Video | googlebot-video | — | not yet seen | |
| Googlebot | googlebot | 1 | 5 hours ago | |
| FeedFetcher-Google | feedfetcher-google | — | not yet seen | |
| GoogleProducer | googleproducer | — | not yet seen | |
| BingVideoPreview | Microsoft | bingvideopreview | — | not yet seen |
| Bingbot | Microsoft | bingbot | 8 | 4 hours ago |
| msnbot | Microsoft | msnbot | — | not yet seen |
| Applebot | Apple | applebot | 2 | 12 hours ago |
| DuckDuckBot | DuckDuckGo | duckduckbot | 1 | 6 hours ago |
| YandexBot | Yandex | yandex | — | not yet seen |
| Baiduspider | Baidu | baiduspider | 1 | 8 hours ago |
| PetalBot | Huawei | petalbot | 30 | 2 hours ago |
| Slurp | Yahoo | slurp | — | not yet seen |
| SeznamBot | Seznam | seznambot | — | not yet seen |
| Sogou web spider | Sogou | sogou | 4 | 3 hours ago |
AI · live answer fetch 7
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | chatgpt-user | 2 | 4 hours ago |
| Claude-User | Anthropic | claude-user | — | not yet seen |
| Perplexity-User | Perplexity | perplexity-user | — | not yet seen |
| MistralAI-User | Mistral | mistralai-user | — | not yet seen |
| Meta-ExternalFetcher | Meta | meta-externalfetcher | — | not yet seen |
| Amzn-User | Amazon | amzn-user | — | not yet seen |
| Google-GeminiNotebook | google-gemininotebook | — | not yet seen |
AI · agentic browser 3
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| Google-Agent | google-agent | — | not yet seen | |
| HeadlessChrome | unknown | headlesschrome | 12 | 2 hours ago |
| PhantomJS | unknown | phantomjs | — | not yet seen |
Social link preview 10
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| facebookexternalhit | Meta | facebookexternalhit | — | not yet seen |
| Twitterbot | X | twitterbot | — | not yet seen |
| LinkedInBot | linkedinbot | — | not yet seen | |
| Pinterestbot | pinterestbot | — | not yet seen | |
| Slackbot | Slack | slackbot | — | not yet seen |
| Discordbot | Discord | discordbot | — | not yet seen |
| Meta | whatsapp | — | not yet seen | |
| TelegramBot | Telegram | telegrambot | — | not yet seen |
| GoogleMessages | googlemessages | — | not yet seen | |
| MicrosoftPreview | Microsoft | microsoftpreview | — | not yet seen |
Ad-quality crawl 5
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| AdsBot-Google-Mobile | adsbot-google-mobile | 2 | 12 hours ago | |
| AdsBot-Google | adsbot-google | — | not yet seen | |
| Mediapartners-Google | mediapartners-google | — | not yet seen | |
| AdIdxBot | Microsoft | adidxbot | — | not yet seen |
| Meta-ExternalAds | Meta | meta-externalads | 2 | 7 hours ago |
Site diagnostics 2
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| Google-InspectionTool | google-inspectiontool | — | not yet seen | |
| Google-Site-Verification | google-site-verification | — | not yet seen |
SEO tooling 6
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| SemrushBot | Semrush | semrushbot | 1 | 17 hours ago |
| AhrefsBot | Ahrefs | ahrefsbot | — | not yet seen |
| DotBot | Moz | dotbot | 1 | 41 minutes ago |
| MJ12bot | Majestic | mj12bot | 4 | 2 hours ago |
| DataForSeoBot | DataForSEO | dataforseobot | 1 | 6 hours ago |
| SERankingBacklinksBot | SE Ranking | seranking | — | not yet seen |
Monitoring 4
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| UptimeRobot | monitoring | uptimerobot | — | not yet seen |
| Pingdom | monitoring | pingdom | — | not yet seen |
| StatusCake | monitoring | statuscake | — | not yet seen |
| Datadog | monitoring | datadog | — | not yet seen |
Web archive 2
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| ia_archiver | Internet Archive | ia_archiver | — | not yet seen |
| archive.org_bot | Internet Archive | archive.org_bot | — | not yet seen |
Other bots 6
| Agent | Operator | Identifies as | 24h | Last seen |
|---|---|---|---|---|
| OAI-AdsBot | OpenAI | oai-adsbot | — | not yet seen |
| APIs-Google | apis-google | — | not yet seen | |
| Google-Safety | google-safety | — | not yet seen | |
| Google-Read-Aloud | google-read-aloud | — | not yet seen | |
| Google-CWS | google-cws | — | not yet seen | |
| Google-Pinpoint | google-pinpoint | — | not yet seen |