I just completed a migration from Windows server 2008 R2 to Ubuntu 10.04. I manage a java application (Java 6, Tomcat) that is having some performance issues. I'd like to use JMX to try and troubleshoot, but I can't seem to get jvisualvm to connect.
If I do a ps -ef | grep "java", I see the following parameters.
-Dcom.sun.management.jmxremote.port=8084
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Netstat shows that port 8084 is listening on 0.0.0.0. In my config, JMX is setup to bind to the FQDN of the server (we use a private DNS server). My firewall (IPTABLES/UFW) is setup to allow all outgoing traffic, and to allow incoming traffic on port 8084.
The server itself is virtual with two NICs, a public and a private. The public NIC's gateway is disabled so that connections can only come in on the private side.
When I try to connect jvisualvm to my app server using JMX, I get the following error in jvisualvm.
Cannot connect to [FQDN OMITTED]:8084 using server:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi
If I look in the jvisualvm log, I see the following trace.
NFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi)
java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
Caused: java.io.IOException: Failed to retrieve RMIServer stub
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:451)
[catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:395)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:216)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:205)
at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:267)
at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:185)
at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:283)
at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:261)
at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:577)
at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1030)
Does anyone have any ideas?
You can specify same port for both JMX and RMI conversation:
The problem is that even if JMX is configured to listen on port 8084, after the initial connection is made, the JMX host will dynamically pick a different port for the rest of the "conversation". My firewall allowed traffic through 8084, but was blocking the subsequent traffic.
There are two options.
I struggled with this same issue trying to connect to our new Tomcat8 VMs over JMX (using VisualVM). Our Linux team had to open a ticket with Redhat to help us troubleshoot. But I stumbled on this article before they could resolve our issue.
The solution as mentioned above was to add
and
Xshould match whatever your setting is forComplete List of Args:
Hope this helps someone else having the same issue. Good luck!
Searching for a solution to an identical problem (after solving many many others, ready to pull my hair), I found this thread, and I can share that the solution that worked for me was https://serverfault.com/a/872756/579378 by @MaxRelax -- I would have upvoted his answer but I don't have enough reputation.
Oh, one more thing: Since
-Dcom.sun.management.jmxremote.ssl=falsebe sure to check the 'Do not require SSL connection' checkbox in the Java Visual VM dialog for adding a JMX connection.I have same problem too on tomcat 9.0.9. on
path/to/tomcat/bin/setenv.sh, following configuration worked for me: