fail2banのapache-badbotsがウンともスンとも動かない

それなりにbotっぽいやつからアクセスは来ているんだけど、ウンともスンとも言わない。

fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/apache-badbots.conf

とかしてもまったくヒットしない。。。

以下のページで別のfailregexが紹介されていたので、apache-badbots.confのfailregexを以下のように置き換えてみたところ、大量にBanが走ったが。。。合っているのか？900近くまでいってピタリと止まった。

[FR]: filter for apache bots doesn’t match · Issue #3445 · fail2ban/fail2ban · GitHub

failregex = ^<ADDR> [^"]*"[^"]+" \d+ \d+ "[^"]*" "[^"]*\b(?:<badbots>|<badbotscustom>)\b[^"]*"
↓<HOST>でも動いているっぽいが。。。
failregex = ^<HOST> [^"]*"[^"]+" \d+ \d+ "[^"]*" "[^"]*\b(?:<badbots>|<badbotscustom>)\b[^"]*"

その他参考

Fail2ban to Limit Bots · Issue #2779 · fail2ban/fail2ban · GitHub

Fail2ban apache-badbots regex needed – General system / Applications – EndeavourOS

↓これが真因？

fail2ban – src256 wiki

上記サイトより

failregexは初期状態だと最後のUserAgentに完全一致になっている。".*"を前後に追加して部分一致で良くする(そのかわり過剰にマッチしないか気にしないとイケない)。
[Definition]
#failregex = ^<HOST>.*"(GET|POST|HEAD).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s).*"$
failregex = ^<HOST>.*"(GET|POST|HEAD).*HTTP.*".*(?:%(badbots)s|%(badbotscustom)s).*"$

Question – Fail2ban trying to setup a rule to ban ai bots | Plesk Forum

こちらの例も。
やっぱデフォだとちゃんと動かないっぽくない？

thread Issue - Default plesk-apache-badbot fail2ban doesn't work

failregex = ^<HOST> -[^"]*"(?:GET|POST|HEAD) \/.* HTTP\/\d(?:\.\d+)" \d+ \d+ "[^"]*" "[^"]*(?:%(badbots)s|%(badbotscustom)s)[^"]*"$

載っていたconfがまんま使える。botの新しいリストもあって便利。最近、うちに激しくきていた、AmazonBotとClaudeBotもある。

[Definition]
badbotscustom = thesis-research-bot
badbots = 80legs|360Spider|anthropic-ai|CCBot|claudebot|ClaudeBot|Claude-Web|ChatGPT|GPTBot|HTTrack|acunetix|adscanner|ag_dm_spider|aiHitBot|Ahrefs|AhrefsBot|Alibaba|alibababot|ALittle|Amazon|amazonbot|AmazonBot|applebot|Applebot|BacklinkCrawler|baidu|Baiduspider|Barkrowler|babbar|BLEXBot|BUbiNG|Buck|Bytespider|Bytedance|chimebot|Cliqzbot|clshttp|Cohere|cohere-ai|CommonCrawl|coccoc|coccocbot|coccocbot-image|DataForSeoBot/1\.0|DiffBot|DigExt|domaincrawler|DomainCrawler|DomainRe-AnimatorBot|domaintools|DotBot|Exabot|extract|Ezooms|GarlikCrawler|ChatGPT-User|ggpht|Google Extended|Google-Extended|Gosign-Security-Crawler|grab|gumgum-bot|FacebookBot|facebookexternalhit|fidget-spinner-bot|fr-crawler|harvest|HaosouSpider|JobboerseBot|jobs.de-Robot|ICCrawler|Imagesift|ImagesiftBot|IndeedBot|Keybot|Kraken|LamarkBot|LieBaoFast|Linguee|LinkpadBot|LinkStats|Lipperhey-Kaus-Australis|ltx71|magpie-crawler|majestic12|Mb2345Browser|meanpathbot|MegaIndex|MegaIndex\.ru|MetaJobBot|MJ12|MJ12Bot|mj12bot|mindUpBot|miner|MQQBrowser|netEstate|nikto|oBot|Omgili|Omgilibot|OpenHoseBot|openlinkprofiler|opensiteexplorer|Paqlebot|paqlebot|PerplexityBot|petalbot|petalsearch|petalsearchBot|PhantomJS|Plista|plukkie|postmanruntime|python-requests|Qwantify|SabsimBot|SafeDNSBot|scrapy|ScreamingFrogSEOSpider|SearchmetricsBot|seek|SeekportBot|Semrush|SemrushBot|SemrushBot-BA|SemrushBot-SA|serpstatbot|SISTRIX|Sistrix|sentibot|seocompany|SEOdiver|SEOkicks|SEOkicks-Robot|seoscanners|seznam|SeznamBot|sg-Orbiter|Siteliner|Snap|sogou|spbot|spot|Squigglebot|SquigglebotBot|ssearch_bot|SurveyBot|R6_CommentReader|RestSharp|rogerbot|TalkTalk|ThumbSniper|trendictionbot|trendkite-akashic-crawler|turnitinbot|TwengaBot|UCBrowser|um-IC|UnisterBot|Uptimebot|VelenPublicWebCrawler|VoidEYE|WBSearchBot|webcrawl|webprosbot|winhttp|wotbox|yandex|YandexBot|YottaShopping_Bot|YouBot|ZoominfoBot|Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailCollector|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 \+http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|(?:Mozilla/\d+\.\d+ )?Jorgee|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots&#44; \+http\://www\.snap\.com\)|Sogou|sogou develop spider|sogou music spider|Sogou Orion spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|TrackBack/1\.02|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|WebEMailExtrac|Wells Search II|WEP Search 00
failregex = ^<HOST> -[^"]*"(?:GET|POST|HEAD) \/.* HTTP\/\d(?:\.\d+)" \d+ \d+ "[^"]*" "[^"]*(?:%(badbots)s|%(badbotscustom)s)[^"]*"$
ignoreregex =
datepattern = ^[^\[]*\[({DATE})
{^LN-BEG}