ES Version: 1.6.2
We have a case where we index documents into timebased indices in a certain pattern: events-2016.10.13 for example.
We have 10 data nodes and 30 shards with 2 replicas and we are using a routing key to index and query documents.
I noticed a weird behavior where the same shard number is always allocated to the same node, so for example for today's index, shard number 1 is allocated to node number 1 and the same is true for yesterday's index and the one before etc..
Is this the normal behavior? Since we're using a routing key, and the same documents are always indexed to the same shard number (according to the docs, the calculation should always yield the same shard number to each routing key), and considering the above behavior - this means that when we run a query with a certain routing key for 30 days back, all the work is being performed by the same node rather than using the entire cluster. Same goes for indexing and so forth.
I also noticed that if we restart the nodes, they are being spread around in a different manner across the nodes.
Is there a way to tell Elasticsearch NOT to place shard number 1 always on node 1?