That’s a short, but accurate, summary of the message I’ve received from Google shortly after they lost control of their crawlers and shut down one of my servers. As it turns out, the default is for their bots to consume up to unlimited resources while indexing a site’s content. Google has also declared themselves free to probe a site using any means at their disposal (via their terms of service, which you’ve “agreed” to unless you’ve opted your sites out of their index), including massive POST queries for which there are exactly zero legitimate reasons (dozens of megabytes, repeated thousands of times). Sound familiar? It should, it’s one of several tools malicious hackers use to conduct denial-of-service attacks.
It gets better. There’s no way I can fight Google on this without gobs of money and a lot of free time, so I gave in and signed into their Webmaster Tools. After adjusting the preferences and reducing the size of the stick their search bots are allowed to carry when visiting, the following message popped up:
To: Webmaster of https://adventurist.life/,
Google Search Console has detected that you or a verified site owner of https://adventurist.life/ changed the maximum crawl rate setting for your website. Within a day or two, Google crawlers will change crawling to the maximum rate that you set. Because crawl rates impact how quickly users can see your updated content in search results, we recommend not limiting Google’s crawl rate. Only do so if your server experiences severe traffic load problems.
New crawl rate: 0.031 requests per second
Status: Begin within 2 days, effective for 90 days.
So not only am I being extorted by a company that’s too big for the rule of law, and forced into setting up an account under the threat of a server-melting denial-of-service attack, I’m also required to hop back on their system every three months and beg them to let my property alone? If I threatened to burn down Google’s data center for not subscribing to this blog, at best I’d get thrown in prison. But, because they are big bad motherfucking Google, I’m at their mercy?
Meanwhile my techs and I are out hundreds of man-hours fighting off their malicious attacks, not to mention the loss of business from days of downtime (which just happened to occur at a critically important time). Sure it’s a rare occurrence, it only affects a handful of sites, whatever. Think about it: if I occasionally shut your business down just for the hell of it, what would you do?