Questions[googlebot](server)

In my sites I have created a script that sends me an email every time a new ip claiming to be google visits the site.

When I see the email I go to check (for example on whois.com) if the ip that claims to be google is really google, and if not, I block it with the firewall.

Normally I find one or two fake googles a week, but in the last few days google has been igniting my server.

 103432 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.130 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
1022802 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.80 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
1063366 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
1178083 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

(number of google accesses on my server in the last 24 hours)

Something is happening and google is entering my server much more than usual and along with the google accesses, the "fake google" has increased a lot. But it is strange that they have increased together...

Will I be blocking the IP of some google service?

(These are those of the last 24 hours)

$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.203.11.230' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.80.104.189' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.148.124.171' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='37.44.196.194' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='88.218.45.98' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='46.161.60.168' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.60.21.63' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='85.202.195.178' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.148.124.139' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='84.54.58.80' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.148.234.198' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.119.46.111' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='195.133.24.218' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.142.55.37' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.87.112.182' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.93.195.206' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.156.125.92' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.119.46.82' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.140.206.107' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.66.208.145' reject"
$ sudo firewall-cmd --reload

they seem to come almost all from the same sources (usually whois.com results are different from each other).

So the doubt comes, am I blocking something that is part of google? Like lighthouse, pagespeed or something else that is still part of the google scans? Or are they just the ip of scammers who pretend to be google to hack my server or clone my sites?

Could it be that google related services claim to be google in the HTTP_USER_AGENT, and then there is nothing about google by verifying domain ownership on whois.com? (Even if google itself says that the only way to actually verify that an ip belongs to them is to verify its ownership, for example with a reverse dns?)

Can you help me understand who they are, and what should I do with these IPs?

Thanks

----------------------update------------------

Perhaps we are going a little off topic, perhaps because some information is missing.

• I manually check every single ip that claims to be google and if it is not google (in dns) I block it with the firewall. I don't use an automatic reverse proxy, because it is actually too long and heavy.

• I use fail2ban, I have many filters that block many scam ip's, these are the ones that survive my fail2ban filters.

• I don't want to block or hide my content, I want it to be seen and understood by search engines, but possibly not cloned entire sites.

• The "fake google" IPs that I usually find on my server belong to sql injection attempts, inserting advertising comments on my sites, and unfortunately in the past I have also found entire sites cloned. (later I added some tricks that prevent cloning).

From further checks it appears that these new "fake google" IPs all visit the same site on my server, each on a different page, and enter my server only once.

Once, I think that's why they manage to survive my fail2ban.

Everything leads to think of an attempt at cloning, even if it is strange that it is done with all these ip (no?).

But the thing that most gives me to think is the fact that the intrusions by these "fake google" are exponentially increased together with the scans of the "real google" IPs.

And this makes me think that somehow they are connected, even if I can't find a connection between real and fake google IPs.

The help I would like from you is to actually understand who they are? And in case they are just scammers, how to block them.

These are today's IPs:

$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='85.202.194.0/24' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='85.202.194.214' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='62.76.232.248' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.193.13.168' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.104.11.51' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.58.68.26' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='5.133.123.198' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.140.207.224' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.192.28.155' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.142.52.178' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.89.100.216' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.93.192.236' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='195.133.24.190' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.140.206.79' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.68.184.215' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.250.46.119' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='31.40.249.208' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.233.187.174' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='37.44.253.226' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='93.177.118.162' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='37.44.253.229' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='94.158.22.50' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.171.226.158' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.250.45.51' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.193.15.194' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.99.26.102' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.142.55.48' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.14.194.78' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='77.220.193.69' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.203.10.224' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.202.82.135' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.31.126.107' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.142.54.110' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='77.220.194.114' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.151.189.82' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.60.21.72' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='195.133.24.68' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.250.45.43' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.88.100.212' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.193.14.102' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.80.104.253' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.58.68.204' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.58.33.128' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.68.247.216' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.171.227.121' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.192.28.169' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='77.243.91.234' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.87.116.145' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.58.33.192' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='45.80.104.160' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='91.222.239.251' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.87.52.175' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.68.185.126' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.31.126.78' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.119.46.226' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='141.98.87.44' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.99.26.51' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='31.40.248.142' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='212.193.14.15' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.171.252.202' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='193.124.9.221' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='194.58.34.69' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='83.142.52.224' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='77.220.193.239' reject"
$ sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.233.187.46' reject"

I was alerted by my Plesk server that an IP Address had been banned. Normally I don't check banned IPs, but this one happened to coincide with our site going down for 1 minute at the same time.

Banned the following ip addresses on Mon Jul 27 21:05:01 AEST 2020
216.239.38.21 with 154 connections

I use the Web Application Firewall (ModSecurity) that plesk provides

A quick check tells me it is a Google IP: https://whatismyipaddress.com/ip/216.239.38.21

Hostname:   any-in-2615.1e100.net
ASN:    15169
ISP:    Google
Organization:   Google

However, Google have instructions on how Verifying Googlebot

Example 1:

> host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

> host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

I also thought "154 connections sounds malicious", but according to Google's own Change Googlebot crawl rate, it shows an example of 5 per second, which would be 300 a minute

The term crawl rate means how many requests per second Googlebot makes 
to your site when it is crawling it: for example, 5 requests per second.
    
You cannot change how often Google crawls your site, but if you want
Google to crawl new or updated content on your site, you can request a recrawl.

After running nslookup -type=ptr 216.239.38.21 I get the same hostname as above, which resolves to a spammy Google Blogger looking website.

So, the IP address is Googles, but it's a spammy looking Blogger.com website, so does that mean it was malicious or a false positive?

The fact that the hostname is any-in-2615.1e100.net leads me to beleive it's a fairly sophisticated spoofed IP address, seems bizarre. I was hoping someone with more technical experince might have futher insights.

We run a number of LAMP servers on AWS with a few dozen websites on them, that customers pay us to design, build and host. They're Ubuntu 14.04 servers with Varnish, Apache and PHP.

Currently, if a customer wanted to have SSL/TLS for their website, we'd put an Amazon ELB load-balancer in front of the server to offload the TLS-connection so Varnish could still cache the content. So, each server was eventually fronted by a half dozen ELBs (one for each TLS customer or site) while Non-TLS sites are handled directly by the server.

To reduce cost and make setup easier we want to eliminate all the ELBs and terminate all TLS connections directly on the servers. This can easily be achieved by running a reverse-proxy in front of Varnish, with Let's Encrypt and SNI. Something like Hitch, Traefik or Nginx.

Some sites are not ready for TLS yet. They require work to fix mixed-content warnings and prevent a SEO drop and not all customers have the budget.

I can open up port 443 on a server and run a reverse-proxy with TLS certificates installed for all 'ready' sites. Clients will unfortunately still be able to connect to 'not ready' sites, though they will get certificate errors (Common Name mismatch, self-signed, etc.). We don't intend to link to the HTTPS versions of 'not ready' sites of course, but someone could still type https://.

I want to prevent a loss in SEO ranking for all sites, mainly in Google. I've been warned that Googlebot will discover the HTTPS-version of 'not ready' sites and index them despite the certificate errors and despite them not being advertised as HTTPS. This would lead to a horrible experience for visitors of those links as well as a serious ranking loss. SEO ranking are hard to gain but easy to lose.

How does Googlebot (and perhaps similar bots) deal with HTTPS versions of 'not ready' sites? Will they be indexed, despite being broken and not being advertised?

How do you mitigate unwanted side effects when partially enabling HTTPS via SNI?

I have some backend url that I use for myself in google chrome only. It's not open public. However for some reason, this bot "Google Favicon" ip located at Google call this URL which I do not want. My guess is Google get this URL from my Google Chrome and try to update cache to this URL everyday. What should I do? I'm not sure if I block its ip, there will be called from new ip later or not.