Add-ons Known Bots 6.1.0

• This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.
• If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.
• After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root:

php cmd.php known-bots:fetch -f

KnownBots v4 is a completely new build - bots are no longer hard coded, but updated via API calls and uses the XF code cache to store bot data

• raw bot data downloaded from API is stored in internal_data/knownbots.json
• new CLI tool for manually fetching bots from API (Cron task is also provided)
• new CLI tool for manually loading bots from knownbots.json
• new CLI tool for testing user agent matches

new bots discovered in June 2021

• btbot: BT Bot
• catchbot: Catchbot
• comodospider: Comodo SSL Spider
• deepnoc: deepnoc bot (network optimized crawling)
• dispenserbot: Dispenser Dab Solver Checklinks Bot
• epicbot: Epictions EpicBot
• esperanzabot: EsperanzaBot
• fast enterprise crawler: Fast enterprise crawler 6 used by Schibsted
• fleabot: Mercadopar Fleabot
• fuseonbot: Fuseon Link Affinity Bot
• google/bot: google/bot
• greenflare seo crawler: Greenflare SEO Crawler
• gsitecrawler: GSiteCrawler
• holmes: Morfeo Holmes Bot
• infotigerbot: InfoTiger Search Engine Bot
• internet security survey bot: Internet Security Survey Bot
• jetpack-bot: JetPack Bot
• mojoo robot: Mojoo Bot
• nicecrawler: NiceCrawler
• prem.moe crawler: Prem.moe Crawler
• sayindexbot: SayIndex Bot
• sbl-bot: SoftByte Labs Bot
• siteliner: Siteliner Bot
• swjschecketbot: Swjschecketbot
• trade desk ads.txt & sellers.json crawler: Trade Desk ads.txt & sellers.json crawler
• vortex: Marty Anstey Vortex Bot
• www.hlabs.co.ke: hLabs Bot
• zspider: Red Kolibri ZSpider