new bots discovered in June 2021
- btbot: BT Bot
- catchbot: Catchbot
- comodospider: Comodo SSL Spider
- deepnoc: deepnoc bot (network optimized crawling)
- dispenserbot: Dispenser Dab Solver Checklinks Bot
- epicbot: Epictions EpicBot
- esperanzabot: EsperanzaBot
- fast enterprise crawler: Fast enterprise crawler 6 used by Schibsted
- fleabot: Mercadopar Fleabot
- fuseonbot: Fuseon Link Affinity Bot
- google/bot: google/bot
- greenflare seo crawler: Greenflare SEO Crawler
- gsitecrawler: GSiteCrawler
- holmes: Morfeo Holmes Bot
- infotigerbot: InfoTiger Search Engine Bot
- internet security survey bot: Internet Security Survey Bot
- jetpack-bot: JetPack Bot
- mojoo robot: Mojoo Bot
- nicecrawler: NiceCrawler
- prem.moe crawler: Prem.moe Crawler
- sayindexbot: SayIndex Bot
- sbl-bot: SoftByte Labs Bot
- siteliner: Siteliner Bot
- swjschecketbot: Swjschecketbot
- trade desk ads.txt & sellers.json crawler: Trade Desk ads.txt & sellers.json crawler
- vortex: Marty Anstey Vortex Bot
- www.hlabs.co.ke: hLabs Bot
- zspider: Red Kolibri ZSpider
new false positives:
new bots:
- spider v9 phone
- 80legs.com: 80legs Crawler
- adssellerscrawler: Ads Sellers Crawler
- apesearch crawler: ApeSearch Crawler
- aportcatalogrobot: AportCatalogRobot
- bot dns-cache.fr: bot dns-cache.fr
- crusty broad web crawler: Crusty Broad Web Crawler
- docomo: NTT Docomo Goo Bot
- dubbotbot: DubBot Bot
- electricmonk: DueDil Electric Monk Crawler
- finbot: Finbot
- flok's crawler: Flok's Crawler
- goodbot: GoodBot
- grover: Grover web crawler
- joc web spider: JOC Web Spider
- linkarchiver twitter bot: LinkArchiver Twitter Bot
- linkscrawler: LinksCrawler
- neevabot: Neevabot Search Engine Bot
- newsharecounts.com: Newshare Counts
- newslitbot: Newslit Bot
- opensearch@mpdl: OpenSearch@MPDL
- orbbot: orbbot
- pajbot1: Pajbot
- richaudience brandsafety bot: Rich Audience Brandsafety Bot
- semjibot: SemjiBot
- simplecrawler/0.1: SimpleCrawler
- snap url preview service: Snap URL Preview Service
- spider_bot/3.0: Spider_bot
- spotibobot: Spotibo bot
- synologychatbot: Synology Chat Bot
- terrawizbot: TerrawizBot
- top100.rambler.ru crawler: Top-100 Rambler Crawler
- webshopchecker bot: Webshop Checker Bot
- wi job roboter spider: Web Integration Job Robot
- woobot: WooBot
- woriobot: Zite.com WorioBot
- wp fastest cache preload bot: WP Fastest Cache Preload Bot
- xenforo: XenForo
- your bot: Your Bot
- yunsecuritybot: YunSecurityBot
- yurichevbot: Yurichev Bot
ew false positives:
new bots:
- cubot; j5
- baiduboxapp
- 200pleasebot
- a8bot
- abilogicbot
- acoonbot
- adform robot
- arhpostbot
- atomseobot
- awariorendererbot
- badoobot
- bl.uk_ldfc_bot
- brobot
- charityengine bot
- charlotte
- cosmos
- coveobot
- crawlbot/1.0.0
- cxensebot
- facebot
- fandomopengraphbot
- freshpingbot
- fuelbot
- geedobot
- getlocalbot
- google-safety
- gpcsupbot
- grub-client
- gynxbot
- healrworld crawler
- hgfalphaxcrawl
- hoodle crawler
- idmarch automatic
- imrbot
- jambot
- justlocal.nl
- kantarsifomediaauditbot
- keobsbot
- keybasebot
- koepabot
- lanaibot
- landsbokasafn
- lapozzbot
- linkpulse metacrawler
- linksmanager.com_bot
- lxrbot
- mbot v
- moreoverbot
- netpeakspiderbot
- www.niraiya.com
- node/simplecrawler
- nu.marginalia.wmsa.edge-crawler
- nutchcvs
- oer commons bot
- omniexplorer_bot
- onefuncbot
- oozbot
- pickybot
- piepmatz bot
- plukkie
- pu_in crawler
- punkspider
- pwa-crawler
- reasonalbot
- revuebot
- runet-research-crawler
- screenerbot crawler
- searchenginecrawler
- sebot-wa
- seekbot
- shopwiki
- showyoubot
- siteauditbot
- sitescorebot
- spinn3r
- squirrobot
- ssl-crawler
- thinkbot
- tsmbot
- tweetedtimes bot
- ucrawl
- umichbot
- urlappendbot
- verticalleap-sitestatusbot
- webgraph
- weblinkchecker
- websquash.com
- wellknownbot
- wizenozebot