Elasticsearch 1.5.2 shared allocation stuck

Elastic Stack Elasticsearch
#1

I have a cluster with the following specs (es v 1.5.2):

3 nodes with RAM: 32GB, CPU cores: 8 each
63 total indices = 32 marvel + 1 kibana + 30 data
366 total shards = (32 marvel + 1 kibana + 150 data)* 1 replica
959,231,444 total docs
588.38GB total data
ES_HEAP_SIZE=16g

I have successfully deleted around 200 empty indices and restarted the cluster (didnt disable shard allocation)- normally the allocation take 1 hour to finish but now it is over 12 hours and still I have 183/366 unassigned shared (half!).

Also, I can see that node1 have only 6 shards allocated to it- most of the data shards is separated between node2 and node3. I tried to restart node1 and still got the same situation. why isn't it taking more shards to it? And why the allocation is stuck?

#2

Ok, I find a solution- I just update indices settings by setting the number of the replica for all of the indices to 0, then the cluster health turned green and finally updated the settings again by setting the number of the replica for all of the indices back to 1.

The cluster started to allocate the shards normally (to node1 also).

#3

You should really upgrade, 1.5 has a number of known issues.

#4

We are also at 1.5.X. Where can I find the list of known issues?

#5

We don't maintain that, by known issues I mean we knew there were problems.

Things like allocation improvements in 1.6. From memory, there was a few corruption issues too.

#6