Some of you were curious. Text assistants and bots are booming, as shovelers seek the god oracle and magnum opus. Here’s the numbers for a naked robots.txt. Inferences, up to you.
In (30 day) levels of aggression, with the most aggresive/inefficient crawler at the bottom;
- Applebot/0.1 [20.1 ]
- heritrix/3.4.0-20200304 [21.8 MiB]
- YandexBot/3.0 [28 MiB]
- GPTBot/1.0 [30.1 MiB]
- IonCrawl [31.6 MiB]
- CCBot/2.0 [40.6 MiB]
- DataForSeoBot/1.0 [51.2 MiB]
- SeznamBot/4.0 [58.6 MiB]
- facebookexternalhit/1.1 [64.1 MiB]
- Google-Read-Aloud [86 MiB]
- PetalBot [91.3 MiB]
- Barkrowler/0.9 [98.6 MiB]
- SemrushBot/7~bl [141.1 MiB]
- BLEXBot/1.0 [152 MiB]
- AhrefsBot/7.0 [159.5 MiB]
- MJ12bot/v1.4.8 [285.2 MiB]
- bingbot/2.0 [373.4 MiB]
- Googlebot/2.1 [535.2 MiB]
- Bytespider [543 MiB]
- DotBot/1.2 [806 MiB]
- Amazonbot/0.1 [2.6 ]