Well it was 7 months ago this happened so this is most likely solved already. But I'll answer just so this question isn't left unanswered.
There is little info about your environment. So there will be a few "if" here...
First off if it's an old Couchbase like 1.8.1, then I can say that I've experienced similar issues.
One thing is that the rebalance does the first 80% a lot quicker than the later parts. It was that for me with Couchbase 1.8.1. But the rebalance keept failing and ultimately we had backups and dumped the entire database and started fresh from backups. We decided that it was worth the downtime.
And later on I did an upgrade to 2.5.1. A lot have improvements to the rebalancing part have happened.
As for diagnosing. You would have to look in the log files and see if you can find any clues. If there were any errors during rebalance. (You can see the logs from Couchbase WebUI..)
So this is my answer:
Check logs. Google to see if there is an bug or an issue with a workaround.
If that fails and rebalance doesn't appear to work. Swollow your pride and accept downtime during a restore from backup (make sure you have tested backup/restore before doing it on a production system).
If running an old version of Couchbase. Perhaps it's time to add a node with a new version and do a online upgrade if possible.
And remember Couchbase is a open source project. So you can compile from source if you want to run the same version as Enterprise Edition. Or if you know a bug has been fixed in something that is newer than the "Community Edition" currently on Couchbase webpage.
Well it was 7 months ago this happened so this is most likely solved already. But I'll answer just so this question isn't left unanswered.
There is little info about your environment. So there will be a few "if" here...
First off if it's an old Couchbase like 1.8.1, then I can say that I've experienced similar issues. One thing is that the rebalance does the first 80% a lot quicker than the later parts. It was that for me with Couchbase 1.8.1. But the rebalance keept failing and ultimately we had backups and dumped the entire database and started fresh from backups. We decided that it was worth the downtime.
And later on I did an upgrade to 2.5.1. A lot have improvements to the rebalancing part have happened.
As for diagnosing. You would have to look in the log files and see if you can find any clues. If there were any errors during rebalance. (You can see the logs from Couchbase WebUI..)
So this is my answer:
And remember Couchbase is a open source project. So you can compile from source if you want to run the same version as Enterprise Edition. Or if you know a bug has been fixed in something that is newer than the "Community Edition" currently on Couchbase webpage.