Home / server / Questions / 959385
In Process
MeanwhileInHell
Asked: 2019-03-22 09:58:10 +0800 CST

Connection refused with internal private DNS name

  • 772

I have a Cassandra cluster set up on 5 EC2 nodes. The nodes can communicate using the private IP addresses just fine, but get connection failures when using the private internal DNS name. Localhost also fails.

I am trying to use the cassandra-stress tool but I get the Failed to connect over JMX; not collecting these stats, and when turning on verbose logging I get the following stack:

java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: ip-172-1-2-2.eu-west-1.compute.internal; nested exception is:
    java.net.ConnectException: Connection refused (Connection refused)]
    at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99)
    at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:85)
    at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:62)
    at org.apache.cassandra.stress.StressAction.run(StressAction.java:198)
    at org.apache.cassandra.stress.StressAction.runMulti(StressAction.java:126)
    at org.apache.cassandra.stress.StressAction.run(StressAction.java:72)
    at org.apache.cassandra.stress.Stress.main(Stress.java:109)

When I run cqlsh I have to specify the private IP address (./cqlsh 172.1.2.3) as using the default localhost fails.

Is there some sort of additional configuration that is required to map this internal name to localhost, or to get it to resolve the private IP addresses on each node?

EDIT:

Running host ip-172-1-2-3.eu-west-1.compute.internal gives the correct output:

ip-172-1-2-3.eu-west-1.compute.internal has address 172.1.2.3

So it looks like DNS is working correctly.

For my cassandra.yaml settings, I have listen_address commented out, but am using:

listen_interface: eth0
broadcast_address: 172.1.2.3
rpc_address: 172.1.2.3

Replication works fine with these settings. I'm not sure how to configure JMX to listen explicitly on the private IP address.

amazon-web-services

1 Answers

  1. Your command cqlsh 172.1.2.3 fails because you configured rpc_address to listen only on the given address. If you want to listen for CQL traffic everywhere, you need to set rpc_address to 0.0.0.0, and set broadcast_rpc_address to 172.1.2.3 - it's all described in the cassandra.yaml itself.

    Regarding JMX - it's not really required that you had it enabled - it collects some data about garbage collection, etc., that maybe not necessary yet if you're just starting. I recommend to concentrate on testing itself, like, described in this article, for example.

    But if you really want to expose JMX to outside, you need to look into cassandra-env.sh - it has detailed comments on what to do. You only need to make sure that you secured JMX, otherwise anybody can do almost anything with your cluster - wipe data, configure it, etc.

    • 0