+++
date    = "2023-12-29T21:06:09+00:00"
lastmod = "2023-12-29T21:07:11+00:00"
tags    = [ "web" ]
+++

Some of you were curious. Text assistants and bots are booming, as shovelers
seek the god oracle and magnum opus. Here's the numbers for a
[naked robots.txt](https://micro.thedroneely.com/m/robots.txt). Inferences, up
to you.

```mathml { caption="Approximately (at least) 81% of requests are machine generated" }
<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="\frac{7,151,956}{8,800,740} \cross 100 \approx 81.26539359\%">
  <mrow>
    <mfrac>
      <mn>7,151,956</mn>
      <mn>8,800,740</mn>
    </mfrac>
    <mrow>
      <mspace width="0.2222em"></mspace>
      <mo style="font-weight:bold;">×</mo>
      <mspace width="0.2222em"></mspace>
    </mrow>
    <mn>100</mn>
    <mo>≈</mo>
    <mn>81.27</mn>
    <mi>%</mi>
  </mrow>
</math>
```

{{< disclose >}}
In (30 day) levels of aggression, with the most aggresive/inefficient
crawler at the bottom;
{{</ disclose >}}

  1.  [Applebot](https://support.apple.com/en-us/HT204683)/0.1                                               [20.1  {{< abbr MiB "Mebibyte, binary megabytes" >}}]
  2.  [heritrix](http://crawler.archive.org/index.html)/3.4.0-20200304                                       [21.8  MiB]
  3.  [YandexBot](https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html?lang=en)/3.0  [28    MiB]
  4.  [GPTBot](https://platform.openai.com/docs/gptbot)/1.0                                                  [30.1  MiB]
  5.  [IonCrawl](https://www.ionos.de/terms-gtc/faq-crawler-en/)                                             [31.6  MiB]
  6.  [CCBot](https://commoncrawl.org/ccbot)/2.0                                                             [40.6  MiB]
  7.  [DataForSeoBot](https://dataforseo.com/dataforseo-bot)/1.0                                             [51.2  MiB]
  8.  [SeznamBot](https://napoveda.seznam.cz/en/seznambot-crawler/)/4.0                                      [58.6  MiB]
  9.  [facebookexternalhit](https://www.facebook.com/externalhit_uatext.php/)/1.1                            [64.1  MiB]
  10. [Google-Read-Aloud](https://developers.google.com/search/docs/crawling-indexing/read-aloud-user-agent) [86    MiB]
  11. [PetalBot](https://webmaster.petalsearch.com/site/petalbot)                                            [91.3  MiB]
  12. [Barkrowler](https://www.exensa.com/crawl/)/0.9                                                        [98.6  MiB]
  13. [SemrushBot](https://www.semrush.com/bot/)/7~bl                                                        [141.1 MiB]
  14. [BLEXBot](http://www.webmeup-crawler.com/)/1.0                                                         [152   MiB]
  15. [AhrefsBot](https://ahrefs.com/robot)/7.0                                                              [159.5 MiB]
  16. [MJ12bot](https://mj12bot.com/)/v1.4.8                                                                 [285.2 MiB]
  17. [bingbot](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0)/2.0              [373.4 MiB]
  18. [Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot)/2.1                 [535.2 MiB]
  19. [Bytespider](https://www.bytedance.com/en/)                                                            [543   MiB]
  20. [DotBot](https://moz.com/help/moz-procedures/crawlers/dotbot)/1.2                                      [806   MiB]
  21. [Amazonbot](https://developer.amazon.com/support/amazonbot)/0.1                                        [2.6   {{< abbr GiB "Gibibyte, binary gigabytes" >}}]