Some of you were curious. Text assistants and bots are booming, as shovelers seek the god oracle and magnum opus. Here’s the numbers for a naked robots.txt. Inferences, up to you.
<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="\frac{7,151,956}{8,800,740} \cross 100 \approx 81.26539359\%"> <mrow> <mfrac> <mn>7,151,956</mn> <mn>8,800,740</mn> </mfrac> <mrow> <mspace width="0.2222em"></mspace> <mo style="font-weight:bold;">×</mo> <mspace width="0.2222em"></mspace> </mrow> <mn>100</mn> <mo>≈</mo> <mn>81.27</mn> <mi>%</mi> </mrow> </math> In (30 day) levels of aggression, with the most aggresive/inefficient crawler at the bottom; Applebot/0.1 [20.1 MiB] heritrix/3.4.0-20200304 [21.8 MiB] YandexBot/3.0 [28 MiB] GPTBot/1.0 [30.1 MiB] IonCrawl [31.6 MiB] CCBot/2.0 [40.6 MiB] DataForSeoBot/1.0 [51.2 MiB] SeznamBot/4.0 [58.6 MiB] facebookexternalhit/1.1 [64.1 MiB] Google-Read-Aloud [86 MiB] PetalBot [91.3 MiB] Barkrowler/0.9 [98.6 MiB] SemrushBot/7~bl [141.1 MiB] BLEXBot/1.0 [152 MiB] AhrefsBot/7.0 [159.5 MiB] MJ12bot/v1.4.8 [285.2 MiB] bingbot/2.0 [373.4 MiB] Googlebot/2.1 [535.2 MiB] Bytespider [543 MiB] DotBot/1.2 [806 MiB] Amazonbot/0.1 [2.6 GiB]