i have three nodes which should autoconfig into a cluster but for some reason they dont seem to want to cluster up.
The 3 Hyper-V VM Servers are CENTOS 7 installed as compute with just the tool packages installed. the servers have consecutive IP's in the same subnet there is no firewall enabled - they can all ping each other both by DNS and IP. Ports on all servers seem to be open.
[root@manelasticshard01 ~]# netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 10.1.247.246:9200 0.0.0.0:* LISTEN 993 26721 3913/java
tcp 0 0 10.1.247.246:9300 0.0.0.0:* LISTEN 993 23402 3913/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 20731 1549/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 19136 2441/master
tcp6 0 0 :::22 :::* LISTEN 0 20733 1549/sshd
udp 0 0 127.0.0.1:323 0.0.0.0:* 994 16999 913/chronyd
udp6 0 0 ::1:323 :::* 994 17000 913/chronyd
same output for all aside for IP addresses
NODE 3
"cluster_name" : "PACMAN",
"nodes" : {
"NlrIawYrR1aq79P6F_Q3zA" : {
"name" : "node3",
"transport_address" : "manelasticshard03/10.1.247.244:9300",
"host" : "10.1.247.244",
"ip" : "10.1.247.244",
"version" : "2.2.0",
"build" : "8ff36d1",
"http_address" : "manelasticshard03/10.1.247.244:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3968,
"mlockall" : false
}
}
}
}
NODE 2
"cluster_name" : "PACMAN",
"nodes" : {
"9Sv2k73BTFSgxMDRDdpRgA" : {
"name" : "node2",
"transport_address" : "manelasticshard02/10.1.247.245:9300",
"host" : "10.1.247.245",
"ip" : "10.1.247.245",
"version" : "2.2.0",
"build" : "8ff36d1",
"http_address" : "manelasticshard02/10.1.247.245:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3898,
"mlockall" : false
}
}
}
}
NODE 1
"cluster_name" : "PACMAN",
"nodes" : {
"RborUe8CS_C7ynX_yFWJ6Q" : {
"name" : "node1",
"transport_address" : "manelasticshard01/10.1.247.246:9300",
"host" : "10.1.247.246",
"ip" : "10.1.247.246",
"version" : "2.2.0",
"build" : "8ff36d1",
"http_address" : "manelasticshard01/10.1.247.246:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3913,
"mlockall" : false
}
}
}
}
my config is currently as follows but i have pretty much been through every setting i can find.
cluster.name: PACMAN
node.name: "node3"
# node.master: false
# node.data: true
network.host: manelasticshard03
# http.port: 9230
discovery.zen.ping.unicast.hosts: [manelasticshard01, manelasticshard02]
discovery.zen.ping.multicast.enabled: false
# discovery.zen.ping.unicast.enabled: true
# discovery.zen.fd.ping_timeout: 30s
# gateway.expected_nodes: 3
# gateway.recover_after_time: 5m
# gateway.recover_after_nodes: 2
# discovery.zen.minimum_master_nodes: 2
Does anyone have any ideas as to why the nodes are not clustering, or of any other tests i can do to find out.
UPDATE - if i set discovery minimum master nodes to two in order to force the cluster to search for a second node i get the following output.
[root@manelasticshard01 ~]# curl http://manelasticshard01:9200/_cluster/health?pretty=true
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
Even though i had explicitly allowed the port range elasticsearch was using apparently RHEL/CENTOS firewall does not like elasticsearch.
Turning it off completely resolved the issue.
FAIL>