No idea what's going on here, but added a new Riak node to the cluster and committed the changes. The new node has taken 0% of cluster membership while the first node in the cluster is growing with over 37% membership. Here's the graph:
================================= Membership ==================================
Status Ring Pending Node
-------------------------------------------------------------------------------
valid 37.5% 25.0% 'riak@prod-riak-08'
valid 17.2% 25.0% 'riak@prod-riak-09'
valid 25.0% 25.0% 'riak@prod-riak-10'
valid 20.3% 25.0% 'riak@prod-riak-11'
valid 0.0% 0.0% 'riak@prod-riak-12'
In the meantime, it looks like some data is inaccessible. Any idea what's going on? Using Riak 1.4.8.
Most recent log of riak-12:
2014-06-24 09:00:11.142 [info] <0.347.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE throttle from 10 -> 0 msec/key, based on maximum vnode mailbox size 53 from 'riak@prod-riak-09'
2014-06-24 09:02:41.150 [info] <0.347.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE throttle from 0 -> 10 msec/key, based on maximum vnode mailbox size 319 from 'riak@prod-riak-10'
2014-06-24 09:02:56.152 [info] <0.347.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE throttle from 10 -> 0 msec/key, based on maximum vnode mailbox size 1 from 'riak@prod-riak-10'
Okay turns out it is a weird data balancing issue in Riak. Basically, riak-08 was still recovering from a node that had been removed from the cluster (went down, never came up).
After lowering the
riak-admin transfer-limit 30everything began to normalize itself at a consistent pace. After about an hour Riak then rebalanced the data back to riak-12: