Home / server / Questions / 785605
In Process
Gergely Lukacsy
Asked: 2016-06-23 07:50:31 +0800 CST

Redis: failing failover - sentinels can't elect new master node

  • 772

I've set up four server and sentinel instances on four different virtual machines in a simple master-slave setup. The key replication works fine, and all redis nodes recognized each other (I'm referring to the known-sentinel and known-slave options).

The problem comes up when I'm trying to simulate a failure. The sentinels just keep telling that they can't elect a new master (failover-abort-not-elected):

12481:X 22 Jun 17:20:05.761 # <__IP1__>:51002 voted for a3ba8d8cd0c15c8b186b44ba529b83d3a8424a42 27
12481:X 22 Jun 17:20:14.140 # -sdown master mymaster <__IP2__> 6379
12481:X 22 Jun 17:20:14.140 # -odown master mymaster <__IP2__> 6379
12481:X 22 Jun 17:20:16.567 # -failover-abort-not-elected master mymaster <__IP2__> 6379

Another strange thing that when I query the sentinel's state in redis-cli, I keep getting greater values for the number of available sentinel instances than the actual count (which is of course, cannot be greater then 4 (right?)).

$ redis-cli -p 51001
> info
[...]
master0:name=mymaster,status=ok,address=91.120.59.231:6379,slaves=3,sentinels=8

Also, I keep getting these -dup-sentinel messages when starting up sentinel:

14563:X 23 Jun 11:10:18.015 * -dup-sentinel master mymaster <__IP_MASTER__> 6379 #duplicate of <__IP_SLAVE2__>:51003 or 65db9ddba84433f71aa77c9263807e6abd939d2c

Here's one of the sentinels' config:

daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"

# bind 192.168.1.100 10.0.0.1
# port 26379

port 51004

dir "/var/lib/redis"

sentinel monitor mymaster <__IP_MASTER__> 6379 2
sentinel failover-timeout mymaster 60000
sentinel auth-pass mymaster <___________PASSW___________>
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 55428

# Generated by CONFIG REWRITE
maxclients 4064
sentinel known-slave mymaster <__IP_SLAVE1__> 6379
sentinel known-slave mymaster <__IP_SLAVE2__> 6379
sentinel known-slave mymaster <__IP_SLAVE3__> 6379
sentinel known-sentinel mymaster <__IP_SLAVE1__> 51002 bfea8e89873353d5c1e9ed498b17f298bd6b6082
sentinel known-sentinel mymaster <__IP_MASTER__> 26379 c04fc07022a251e56e81e21dbe018bd471745038
sentinel known-sentinel mymaster <__IP_SLAVE2__> 51003 b5a56eb69abb5893203ecf7b1a7d28ae4aa06ab5
sentinel known-sentinel mymaster <__IP_MASTER__> 51001 be1ff6c9e4a62abea132c24447fd3cb38d221496
sentinel known-sentinel mymaster <__IP_SLAVE1__> 26379 f5a7fd0ee199afc02ffafc0e822363acf5595b1c
sentinel known-sentinel mymaster <__IP_SLAVE2__> 26379 6328962293b7838fa88b9f176b9bc4cc08f2e26d
sentinel current-epoch 26

The log files aren't really informative... is there a way to make it a little bit more verbose? Do you have any idea what am I misssing here?

Thanks in advance!

Bumping post:

Checked firewall settings, it should have been okay, I was still getting the same error when I shut off the firewall completley.

redis-sentinel

2 Answers

  1. Taken from sentinel default config:

    # *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1

    So try to bind your server's IP.

    Gluck!

    • 2

  2. Old sentinels are not removed from configuration. Look at docs redis sentinel

    Execute: SENTINEL RESET mymaster and for a while, new master should be elected.

    • 1