Debugging Shard Allocation

Elastic Stack Elasticsearch
1

Is there any way to ask Elasticsearch why it allocated a shard to a specific node?

I've got a couple of different types of indexes in ES (some with small numbers of documents, others with millions more), and after rebooting various nodes to apply updates and, I've got an imbalanced cluster (all the million+ shards are allocated to one node).
Nothing obvious in the cluster or index settings is standing out as to why it's not allocating to the others. If ES can tell me "Index A, Shard B cannot be allocated to Node X because of [y,z]" then that'd be great.

2

then you will need to read into the code, first the interface https://github.com/elastic/elasticsearch/blob/v1.5.2/src/main/java/org/elasticsearch/cluster/routing/allocation/allocator/ShardsAllocator.java

and then the concrete class, e.g. BalancedShardsAllocator.java https://github.com/elastic/elasticsearch/blob/v1.5.2/src/main/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

and based on what concrete class, you see the logic maybe here https://github.com/elastic/elasticsearch/blob/v1.5.2/src/main/java/org/elasticsearch/cluster/routing/allocation/AllocationService.java#L205

hth

jason

3

You can try doing https://www.elastic.co/guide/en/elasticsearch/reference/master/cluster-reroute.html for one shard and use ?explain to see what ES tells u. This is obviously not solving the initial ask, but can help us get some data from ES if indeed it can't allocate to other nodes.

4

Elasticsearch's allocation weights all shards the same. See the pull request below for some of the reasoning as to why. At some point in the future it'll have more, better ways of tuning this stuff, but for now your best bet is setting index.routing.allocation.total_shards_per_node on the big indexes.

1 Like
5

thanks @nik9000
To ensure even allocation of primaries to all nodes, we are setting a bit higher-number to below setting -

cluster.routing.allocation.balance.primary

This was done to overcome issue where one of our 5 data nodes had all primary shards of all indices. We used guess and trial number, do we see any side-effect of messing up with this number :)?

6

Ah, nice. Thanks for that.