Our cluster has 12 shards 4 nodes, 1 replica.
After a crash, sometimes our cluster does not re balance it self.
Right now, the master node has 8 shards all primary, and one of the nodes
only has 3 shards.
Memory usage on the master is 3x higher than on other nodes because of
that. Is there any way to force a rebalance without taking one of the nodes
down?
I wonder if we are seeing some split brain syndrome here. We took down all
nodes, and restarted them with an interval of 2min between them
PS: more than once, we got into an unrecoverable state when trying to start
all nodes at once, that's why we have some interval now.
Any ideas on why this might be happening?
--
Douh! Forget it, I just found that shards are not balanced at index level.
So we had an small index opened, that was unbalancing the search index.
On Monday, December 17, 2012 2:09:20 PM UTC-5, Vinicius Carvalho wrote:
Our cluster has 12 shards 4 nodes, 1 replica.
After a crash, sometimes our cluster does not re balance it self.
Right now, the master node has 8 shards all primary, and one of the nodes
only has 3 shards.
Memory usage on the master is 3x higher than on other nodes because of
that. Is there any way to force a rebalance without taking one of the nodes
down?
I wonder if we are seeing some split brain syndrome here. We took down all
nodes, and restarted them with an interval of 2min between them
PS: more than once, we got into an unrecoverable state when trying to
start all nodes at once, that's why we have some interval now.
Any ideas on why this might be happening?
--