I have this website that was working fine until recently. Now users are reporting that it's not opening in their iphones and ipads.
Doesn't matter what browser you try on iOS, it just won't work. Also it doesn't open in Safari when browsed on a Mac machine. Other browsers work fine though (Mac OSX only, not iOS).
Here is the server configuration:
DigitalOcean + Ubuntu 18.04 + Nginx + PHP 7.3 (Laravel) + Let's Encrypt SSL + HTTP2 activated
Before I explain further, I wanna mention that using the same type of configuration as above, I have another server which is running fine and is accessible from all devices. They are almost identical (in terms of config and setup) apart from domain name and ip address obviously.
Another tiny extra info which I think is worth mentioning, is that my users are located in Iran.
Here is a list of things that I've checked:
- The domain is accessible using ping command from iOS and Mac.
- The IP address itself is also accessible, and it can be used to browse the site, it will just show a "invalid ssl certificate" warning.
- The DNS that were used for the domain are all ping-able from iOS and Mac.
- If a VPN connection is used, the site can be accessed. Which is really weird because everything is technically reachable, the domain, the ip and the dns.
- Aside from using a VPN, the site can also be accessible if SSL is completely turned off/disabled.
- Other public websites that are using Let's Encrypt SSL are all accessible. So it can't be a Let's Encrypt issue.
Here is what I've tried so far:
- Tons of googling. I've even came across with a user with same type of issue on here and here and here
- I've tried disabling HTTP2.
- I've double checked my firewall settings, and even tried disabling it.
- I've tried disabling GZIP.
- I did a ssl test on ssllabs.com and it shows no problem and it's identical to my working server.
- Tried using
keepalive_disable "safari"
in nginx config - I've tried swapping
ssl_protocols
values, like only accepting TLSv1.3 or dropping TLSv1.0 - Tried using
ssl_session_cache
- Tried changing
ssl_ecdh_curve
toauto
andsecp384r1
- Tried changing
ssl_ciphers
toHIGH:!aNULL:!MD5;
andEECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
- Tried changing
ssl_session_tickets
toon
andoff
- Tried using
Strict-Transport-Security
- Tried commenting
/etc/letsencrypt/options-ssl-nginx.conf
which is used by certbot - Tried different HTTP to HTTPS redirect settings
- I made sure to use private tab when testing to make sure i'm not looking at a sort of faulty cached attempt to reach the site.
- I did service nginx reload & restart everytime I made a change to config files.
- I did lots of
sudo reboots
to make sure it's not a simple reboot issue.
As a final resort, I even went through the trouble of reinstalling the OS from scratch and setting up the server again. Still Doesn't work. And no I didn't copy my config files, I did every command manually and modified config files carefully to make sure I don't bring any problems from previous setup. This also means I did a fresh SSL request from Let's Encrypt (using cerbot). Although I don't know if they generate a new one or the send you the one from last time.
EDIT 1: I want to throw a random thought in here. I really don't know about advanced networking but maybe something from Iran censorship firewalls is meddling with the SSL connection and that causes Apple devices to fail establishing the connection. I heard Apple is strict in it's own way and requires a specific SSL connection unlike chrome and firefox. those 2 might ignore a slight error but Apple devices don't.
EDIT 2: I checked the Safari while Wireshark is tapping. It seems only few packets gets exchanged only no matter what I do. Also it seems Safari is using TLSv1.0 even though I've disabled it in nginx config. I really don't know how completely turn it off so Safari won't even try to use it to communicate with the server.
EDIT 3: I tried using a different SSL (using comodo's 3 month free ssl trial program) and the issue is still there... I read some people saying they resolved some similar issues after changing their SSL. I don't know why it's not working for me.
EDIT 4: I decided to change DNS servers of the domain to see if it's a DNS issue. I don't know why and how but after I changed them, Safari could load the site briefly. But after sometime it stopped working again.
EDIT 5: No luck with disabling external script codes in my site, like google analytics ( because their service is kinda blocked for Iranians ). Again, I read people reports implying that some fixed their issue by disabling external scripts.
EDIT 6: I'm starting to believe this is not a networking or server config issue on my part. It really looks like a 3rd party interfering with the request. I don't know about Apple and how it handles networking, but I think Apple connection/packeting negotiations is causing some sort of red flag so to speak and that makes Iranian censorship/firewall to drop the client connection.
EDIT 7: I even changed the server's datacenter, with a fresh new ip address. the issue still exists :(
EDIT 8:
This is the /var/log/nginx/error.log
output when it's set to debug
. It shows these lines when I initiate a request from a Safari client.
I did try to disable php fpm to make sure it's not a php bottleneck issue. Also I tried to tweak some random parameters, like bumping net.core.somaxconn
to 1024
or increasing backlog
in php fpm config files.
Also when I watch the system using htop
everythink looks normal, no cpu usage spikes and no specific process jumping to top of the list.
EDIT 9: I swapped Nginx to Apache2 to check if it's a webserver issue. Well, it wasn't.
Since you doubt the proxy in-between you and the website, can you bypass it by using the -L option in ssh? It will provide a secure tunnel between you and your website that cannot be altered by the proxy.
For example, execute this on your desktop: ssh -L 2222:127.0.0.1:443 [email protected]
Once this command has logged into your server, fire up the browser on the same desktop to https://127.0.0.1:2222
If you now see you site and its SSL certificate, everything is setup correctly - theres someone in the middle altering your SSL connection when you are not using the tunnel.
I found a temporary solution to my problem. I just moved the server to a different hosting/datacenter!
Update: Actually the problem reappeared again after few months... I really don't know what technical issue is causing the problem or how to fix it.
Update 2: I talked to an IT networking expert and they told me the issue actually relies in government firewall rules picking up on my domain name ( false/wrong censorship ) and it doesn't matter what webserver configuration or webhosting I use, it just keeps getting blocked by firewalls out of my reach.
It could just be the certificate, do you have SubjectAlternativeName properly set? Only using the SubjectName for the DNS name is deprecated. Do you use certificate pinning, OSCP stapling, HSTP, and other relatively new TLS settings? That might cause issues on regulated networks.
Some internet users need to use a proxy with SSL-inspection. Many corporate environments use products similar to IronPort or Bluecoat to detect covert channels and malware as part of their security policy. The way to do this originated from an article in the underground magazine Phrack 76. It boils down to a simple setup where each client has an additional CA root certificate of the proxy that it trusts, the proxy in turn re-initiates connections on behalf of each client, "opening the SSL" by MIT, countries like Iran and China monitor the internet at large. Due to this there is no PFS, and several things might fail that depend on the client checking the servers' crypto, because it is altered or ssl is even completely removed. If you don't care about it then you can downgrade your server certificate settings to "export quality" by removing mentioned options.
@austin-dixon's answer is on the right track. That test would have to be done from the same country the users are in, which may not be an option if you are not already in that country.
You may be able to at least validate the configuration by using the Qualys SSL test at https://www.ssllabs.com/ssltest/. The letter score doesn't matter in this case, just scroll down to Handshake Simulation and see how the Apple devices are expected to perform under normal circumstances.
Either way that just confirms what you already suspect, that there's something in between those users and the site. There are not a lot of ways for a developer to get around that situation short of disabling HTTPS completely. It's possible the man in the middle requires some of the weaker cipher suites or lower TLS versions than the server is configured for, but I wouldn't know which ones to suggest in that case.