I'm trying to get into ReplicaSet concept, and found something weird in mongoDB Documentation:
For a node to be elected primary, it must receive a majority of votes. This is a majority of all votes in the set: if you have a 5-member set and 4 members are down, a majority of the set is still 3 members (floor(5/2)+1). Each member of the set receives a single vote and knows the total number of available votes.
If no node can reach a majority, then no primary can be elected and no data can be written to that replica set (although reads to secondaries are still possible).
(taken from here)
So, If I got that right, in the 5-member case mentioned there the one node that's still standing WILL NOT be chosen as primary and the whole set will not get any writes? and that's even if this single node was the last primary before the elections?
If it's true there can be many less-radical cases which will end up with a degenerated set. How can we avoid this?
The majority rule is intended to prevent multiple single (think isolated) nodes from claiming to be primary at the same time. The strategies to avoid this will be dependent on your architecture.
As some examples - you can have stand by arbiters in each data center or zone so you can add them to form a majority, you can reconfigure the set in the event of a "disaster" to only include 3 nodes instead of 5, thereby letting 2 form a majority etc. Or, you can restart the sole remaining mongod instance (if such a thing exists) as a standalone without replica set options and use it in that manner until you restore your set.
When designing for disaster recovery or failover across multiple zones, the important thing to do is keep the majority rule in mind as you are designing it.