Jordan Rieger's questions

I've got a development web server environment consisting of 2 Windows 2012 VM's configured with NLB (Multicast mode.) They are each using a single network adapter with 2 IP's assigned: the shared NLB IP, and their non-shared individual IP. Our usage pattern is to run builds and deployments to one of the VM's while it is held out of the active NLB pool. Once the build is done, the freshly updated server is added back into the pool, and the other one is removed. This way we maintain constant uptime.

My problem is that on 1 of the 2 servers, IIS will not allow connections from IP addresses outside our network through NLB. So if that server is the only active one in the NLB pool, internal PCs can connect, but hosts outside our network can't connect. They get a timeout. I can see the TCP SYNs using Wireshark, and no ACK ever gets sent back. If I flip the NLB config to the other server, it allows both internal and external IPs to connect. And I verified this from an outside host that I control: I attempted an HTTP request using the NLB IP and got the same result.

Things I've checked:

• As far as I can tell, the NLB and IIS configuration/bindings are the same on both servers.
• There are no IP filtering rules in IIS.
• Windows Firewall is turned off.
• I tried switching to Unicast NLB mode, but that broke everything.
• I tried changing the NLB host priority to 4 and 3 instead of 1 and 2, with no effect.
• I also eliminated host headers and SSL from the equation by reproducing the problem with just a simple request by IP address on port 80.

How can I troubleshoot why IIS is not ACKing the SYN from outside IPs? Can it log this sort of thing?

If I can't troubleshoot why IIS is not allowing the connection on Server 2, my only recourse will be to rebuild the whole VM from scratch and try again...

In troubleshooting an issue with my Tomcat servlet I decided to enable network debugging by editing /var/apache-tomcat-8.5.5/bin/setenv.sh and changing the line export JAVA_OPTS="-Xms512m -Xmx1536m -XX:MaxPermSize=256m" to export JAVA_OPTS="-Xms512m -Xmx1536m -XX:MaxPermSize=256m -Djavax.net.debug=all". As soon as I saved the file and restarted Tomcat (via catalina.sh start/stop) I found that I could no longer connect to the Tomcat manager app (https://10.9.9.236:8443/manager/html/) via HTTPS. Chrome says:

"This site can't be reached" (ERR_CONNECTION_CLOSED)

I reproduced this behavior after completely stopping and un-deploying my webapp. (Only the default Tomcat manager apps are installed now.) So I know it's only that one Java option that is causing HTTPS to fail.

Why would enabling debugging break HTTPS?

Here is a possible clue from the catalina.out log file:

https-jsse-nio-8443-exec-4, fatal error: 80: problem unwrapping net record
java.lang.RuntimeException: java.security.NoSuchAlgorithmException: EC AlgorithmParameters not available
https-jsse-nio-8443-exec-4, SEND TLSv1.2 ALERT:  fatal, description = internal_error
https-jsse-nio-8443-exec-4, WRITE: TLSv1.2 Alert, length = 2
28-Sep-2016 14:01:00.576 SEVERE [https-jsse-nio-8443-exec-4] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun 
 java.lang.RuntimeException: java.security.NoSuchAlgorithmException: EC AlgorithmParameters not available
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1429)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:449)
    at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:227)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1387)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.security.NoSuchAlgorithmException: EC AlgorithmParameters not available
    at sun.security.util.ECUtil.getECParameters(ECUtil.java:100)
    at sun.security.util.ECUtil.getECParameterSpec(ECUtil.java:149)
    at sun.security.ssl.JsseJce.getECParameterSpec(JsseJce.java:385)
    at sun.security.ssl.SupportedEllipticCurvesExtension.toString(SupportedEllipticCurvesExtension.java:127)
    at sun.security.ssl.HelloExtensions.print(HelloExtensions.java:150)
    at sun.security.ssl.HandshakeMessage$ClientHello.print(HandshakeMessage.java:323)
    at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:340)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
    at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:397)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:457)
    ... 7 more
Caused by: java.security.NoSuchAlgorithmException: EC AlgorithmParameters not available
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
    at java.security.Security.getImpl(Security.java:695)
    at java.security.AlgorithmParameters.getInstance(AlgorithmParameters.java:146)
    at sun.security.util.ECUtil.getECParameters(ECUtil.java:98)
    ... 21 more

When I wipe the logs and revert back to the default JAVA_OPTS, the exception does not occur.

Environment:

CentOS Linux release 7.2.1511 (Core)
uname -r: 3.10.0-327.10.1.el7.x86_64

Using CATALINA_BASE:   /var/apache-tomcat-8.5.5
Using CATALINA_HOME:   /var/apache-tomcat-8.5.5
Using CATALINA_TMPDIR: /var/apache-tomcat-8.5.5/temp
Using JRE_HOME:        /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64/jre
Using CLASSPATH:       /var/apache-tomcat-8.5.5/bin/bootstrap.jar:/var/apache-tomcat-8.5.5/bin/tomcat-juli.jar
Using CATALINA_PID:    /var/apache-tomcat-8.5.5/tomcat.pid

I should add that even when HTTPS access to the servlet is broken, I can still access it via HTTP, e.g. http://10.9.9.236:8080/manager/html/.

I have Site A in IIS 8.0 configured to an SSL certificate with the binding ###.###.###.###:443. I also have Site B configured to a different certificate with the binding ###.###.###.###:443 (same IP but host = sub.domain.com) For the Site B binding, I checked the SNI enabled box so IIS allowed it.

The problem is, I find that loading Site B from an SNI-enabled browser serves the certificate for Site A. I assume this is because the Site A binding is technically satisfied, so IIS stops looking at other bindings to serve the request. (I noticed that the netsh http show sslcert command seems to show the Site A binding first in the display order, although I have no idea if that order is meaningful.)

Is there a way to change the binding order in IIS so that it attempts to bind requests to the Site B SNI-host binding first, and only after that falls back to the Site A non-SNI (IP:port only) binding? (For legacy compatibility, I don't want to enable SNI on the Site A binding.)

Suppose I am setting up a new name server at ns1.example.com, and I have properly created the glue record at my registrar to tell the registry that ns1.example.com should point to my name server's IP address at 77.77.77.77. Is there anything in the configuration of my name server that needs to also publish this IP address in a zone? I.e. publish an NS or A record for itself?

If the name server didn't publish its own IP address (aside from in the registry glue record) would clients still be able to query me for other records successfully?

We have setup a Windows Network Load Balancing (NLB) cluster of 2 hosts for our new staging environment. The cluster initially works fine, but eventually, after stopping and starting one of the hosts via our automated deployment script, the first host can no longer see the second host. So if you open NLB Manager on Host1, Host2 is not visible. This doesn't happen if you open NLB Manager on Host2. Edit: Actually, sometimes Host2 cannot see Host1 either. When that happens, the cluster is completely unresponsive to requests.

Things we've noticed during the "bad state":

• The hosts can ping eachother.
• RPC works because I can access the C$ share of one host from another.
• If I try to manually add the missing Host2 to Host1, it says it already exists. I can click Cluster > Connect to Existing and specify Host2, which works, but only until I close NLB Manager and open it again.
• When the cluster is in the bad state, if I try to start Host2, it says "Converging" but never changes to "Converged".

Things we've tried that did not fix the problem:

• Removed all NLB stuff and recreated the configuration from scratch.
• Removed and re-added the network adapter in Device Manager on one of the hosts.
• Switching from Multicast to Unicast.
• Rebuilt the second node's VM from scratch.

Restarting the servers seems to fix it temporarily, until it happens again.

Configuration:

• Both hosts are running Windows Server 2012 R2 with the latest updates as of 2015-09-21. Before the NLB setup, the second host was cloned from an image of the first host.
• Both hosts are running as VMWare guests on the same VMWare host. I'm not sure of the version of VMWare (that's up to our admins) but the VMWare tools on the guest OS's say version 9.4.
• Each host has a single Ethernet adapter with 2 IP's assigned: the host's dedicated IP, and the cluster IP.
• Port rules: Multicast, ports 80 and 443 only, Load Equal, Affinity Single

Background:

My client has an old Pentium III Windows 2003 server whose 16/36 GB disks are dying. On it he has a database-driven web site and email application that needs further customization by a developer (me). First we need to get it working on the new server. The original developer is no longer available to provide a system setup guide. So my client got a tech who imaged the old drives over to the new server and managed to get it booting. But the IIS-driven site no longer works. In fact it seems that IIS itself does not work.

Problem:

Service Unavailable when attempting to browse from the server itself to the URL for a local Web Site called test which I setup in IIS to serve a single static index.htm file. This I did to isolate the problem, and eliminate the client's application from the equation. The site is setup on port 80 with the host header "test.myclientsdomain.com", and I used the etc\hosts file to point that host at the local IP. I know the host entry took effect because I can ping it.

When doing an iisreset, I get:

Attempting start...
Restart attempt failed.
IIS Admin Service or a service dependent on IIS Admin is not active.  It most likely failed to start, which may mean that it's disabled.

Despite this message, the services all stay in the Started state. The only relevant System event logs I found are:

Event Type: Error
Event Source:   W3SVC
Event Category: None
Event ID:   1002
Date:       11/4/2012
Time:       11:04:47 PM
User:       N/A
Computer:   ALPHA1
Description:
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

Event Type: Error
Event Source:   W3SVC
Event Category: None
Event ID:   1039
Date:       11/4/2012
Time:       11:13:12 PM
User:       N/A
Computer:   ALPHA1
Description:
A process serving application pool 'DefaultAppPool' reported a failure. The process id was '5636'.  The data field contains the error number.

Data:
0000: 7e 00 07 80               ~..    

And one Application event log:

Event Type: Error
Event Source:   Windows SharePoint Services 2.0
Event Category: None
Event ID:   1000
Date:       11/4/2012
Time:       11:34:04 PM
User:       N/A
Computer:   ALPHA1
Description:
#50070: Unable to connect to the database STS_Config on ALPHA2\SharePoint.  Check the database connection information and make sure that the database server is running.

That last log tells me that the tech may have initially tried to have both the old and the new server running, by renaming the new server from ALPHA1 to ALPHA2. And perhaps SharePoint grabbed onto that change, and now can't tell that the machine name has been switched back to the old ALPHA1. But why would SharePoint interfere with a static IIS web site serving a single HTML file? The test site is not even within an Application pool (I clicked the Remove button.)

What I have tried/eliminated:

• No relevant services seem to be disabled: IIS Admin, WWW Publishing, Sharepoint Timer
• Giving Full Control to All Users/Everyone on the c:\inetpub\test folder serving my test site.
• I can connect to and query the local SharePoint config database (ALPHA1\SHAREPOINT\STS_CONFIG) from SSMS. But when I try to do stsadm -o setconfigdb -connect -databaseserver ALPHA1\SHAREPOINT it tells me The SharePoint admininstration port does not exist. Please use stsadm.exe to create it. And when I do that, using the port 9487 specified in the IIS SharePoint Admin site config, it tells me the port is already in use. Needless to say, simply browsing to the admin site gives me a similar error about being unable to reach the config database. I didn't want to go further down the SharePoint path as it may be completed unrelated to my IIS issue, and I don't even know yet if SharePoint is required for this application to work. The app itself is ASP.Net/C#/Silverlight and a little MS Word integration (maybe that's where the SharePoint stuff comes in.)

Apache 2.2.3/mod_ssl/CentOS 5.5 VPS

Our certificate expired on 2011-10-06, and even though we have seemingly installed the new one correctly, browsing to the site still shows an expired certificate! I've tried deleting my browser cache and using several different browsers. Relevant lines from the ssl.conf file (I've excluded those commented out.):

Listen 127.0.0.1:443
SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300
# Note - I tried disabling SSLSessionCache with the "none" setting but it didn't help.
<VirtualHost 127.0.0.1:443>
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /var/certs/gentlemanjoe.com/new2011/gentlemanjoe.com.crt
SSLCertificateKeyFile /var/certs/gentlemanjoe.com/new2011/gentlemanjoe.com.key
SSLCertificateChainFile /var/certs/gentlemanjoe.com/new2011/gd_bundle.crt
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
ServerAdmin [email protected]
DocumentRoot /var/www/gentlemanjoe.com
ServerName gentlemanjoe.com
<Directory /var/www/gentlemanjoe.com>
    AllowOverride All
    Order deny,allow
    allow from all
</Directory>          
</VirtualHost>

Things I've Checked

First I tried moving the old cert and key files to a completely different folder to make sure Apache wasn't still grabbing them somehow. Nothing changed. For fun I tried temporarily renaming the new cert and key files, and Apache dutifully complained and refused to start.

Then I tried to make sure I wasn't being fooled by editing the wrong config file. Using "locate" I found only one httpd.conf file under /etc/httpd/conf/httpd.conf. I also used "locate" to verify that there is only one ssl.conf file, /etc/httpd/conf.d/ssl.conf. The key file is what I generated using OpenSSL, following the instructions GoDaddy gave for generating the CSR.

I have verified that I am working with the right site by uploading a test.html file to the /var/www/gentlemanjoe.com folder and verifying that I can browse to it. But if I try to view the test file in HTTPS, I get the same certificate expiration warning.

I verified that the cert itself has the right expiration date:

openssl x509 -in /var/certs/gentlemanjoe.com/new2011/gentlemanjoe.com.crt -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            07:e7:49:69:97:96:16
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certificates.godaddy.com/repository, CN=Go Daddy Secure Certification Authority/serialNumber=07969287
        Validity
            Not Before: Oct 21 17:37:55 2011 GMT
            Not After : Oct  8 21:16:03 2013 GMT
        Subject: C=CA, ST=BC, L=Burnaby, O=Diamond Bailey Consolidated Commercial Services Ltd, OU= , CN=www.gentlemanjoe.com

I tried re-keying the cert at GoDaddy with a new CSR and everything seems to work, but I get the same result in the browser.

Possible Clue #1

Whenever I do "apachectl restart", I see this in the error_log file:

[Fri Oct 21 18:03:33 2011] [notice] SIGHUP received.  Attempting to restart
[Fri Oct 21 18:03:33 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Oct 21 18:03:33 2011] [notice] Digest: done
[Fri Oct 21 18:03:33 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Fri Oct 21 18:03:33 2011] [info] LDAP: SSL support available
[Fri Oct 21 18:03:33 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Fri Oct 21 18:03:33 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Fri Oct 21 18:03:33 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Fri Oct 21 18:03:33 2011] [info] Shared memory session cache initialised
[Fri Oct 21 18:03:33 2011] [info] Init: Initializing (virtual) servers for SSL
[Fri Oct 21 18:03:33 2011] [warn] RSA server certificate CommonName (CN) `www.gentlemanjoe.com' does NOT match server name!?
[Fri Oct 21 18:03:33 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Fri Oct 21 18:03:34 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Fri Oct 21 18:03:34 2011] [info] Server built: Aug 30 2010 12:28:40

The GoDaddy techs tell me that the www vs non-www should not matter, and I tend to agree, since the security warning in my browser is not complaining about a server name mismatch, but rather an expiration, indicating that the old certificate is still being loaded somehow.

Possible Clue #2

The HTTP Server response header for http://gentlemanjoe.com says "Andromeda" rather than "Apache". This seems weird to me since my Googling of "Andromeda" turns up a media-server type project, which wouldn't be installed on this server (but I can't say that with certainty since I didn't set any of this up, the usual admin/developer is on holiday and I'm just helping a friend with his site.) Also, the httpd.conf file does not contain the string "Andromeda" indicating that it hasn't been modified to spit this out. So it might be the Magento e-commerce platform he's using, but what would be the point of replacing the standard Apache response header?