We have a really strange issue with DNS on our EC2 instances. An application we're running on some of our instances is Java-based service that provides a back-end for an Android application. As part of it's functionality, it sends push notifications to phones through a Google API. To do this, it has to make an SSL request to android.apis.google.com. Unfortunately, when running this on our EC2 instances, we get a certificate error because the hostname doesn't match the certificate name:
hostname in certificate didn't match: <android.apis.google.com> != <.gstatic.com> OR <gstatic.com> OR <.gstatic.com>
We have tracked the problem down to a difference in DNS results. When we query DNS for android.apis.google.com from our office (where everything works), we get back the following:
android.apis.google.com. 300 IN CNAME clients.l.google.com.
clients.l.google.com. 160 IN A 74.125.226.230
clients.l.google.com. 160 IN A 74.125.226.231
clients.l.google.com. 160 IN A 74.125.226.232
clients.l.google.com. 160 IN A 74.125.226.233
clients.l.google.com. 160 IN A 74.125.226.238
clients.l.google.com. 160 IN A 74.125.226.224
clients.l.google.com. 160 IN A 74.125.226.225
clients.l.google.com. 160 IN A 74.125.226.226
clients.l.google.com. 160 IN A 74.125.226.227
clients.l.google.com. 160 IN A 74.125.226.228
clients.l.google.com. 160 IN A 74.125.226.229
When we perform the same query from an EC2 server, we get back a different set of DNS results:
android.apis.google.com. 300 IN CNAME clients.l.google.com.
clients.l.google.com. 300 IN A 72.14.204.138
clients.l.google.com. 300 IN A 72.14.204.100
clients.l.google.com. 300 IN A 72.14.204.101
clients.l.google.com. 300 IN A 72.14.204.102
clients.l.google.com. 300 IN A 72.14.204.113
Any ideas why the DNS results would be so dramatically different out at EC2? And, more importantly, how we can fix this?
We did try using a custom hostname validator. According to our developers, that allowed the connection to proceed, but the problem is that it's connected to the wrong server, so the request still fails.
Google has lots and lots of IPs, and they'll serve you the IPs that are geographically closest to where they think you're located.
When I
dig
against Google's own public DNS at8.8.8.8
, I get the same list you get from EC2:The problem you're encountering appears to be this one: https://stackoverflow.com/questions/6296547/c2dm-ioexception-when-sending-message
Strange. Everything started working again - like magic.
It may be a problem coming from Amazon EC2 DNS platform (refresh problem) or a caching results issue coming from both, your "Virtual plateform" or Amazon platform.