ES Version: 1.6.2 We have a case where we index documents into timebased indices in a certain pattern: events-2016.10.13 for example. We have 10 data nodes and 30 shards with 2 replicas and we are using a routing key to index and query documents. I noticed a weird behavior where the same shard nu…

Why are you routing as well as using time based indices?

I need to be able to purge data from old indices, why? Is there a conflict?

Then why not just delete the indices? That's the whole idea of why we suggest them, because it's easier to delete an index as opposed to doing a delete by, or similar.

Not sure I follow. I have time based indices as many others do. I also use routing to save some resources by using routing so that I can enjoy some extra performance boost by storing all documents for a specific client in a single shard on the index. [image] warkolm: Then why not just delete th…

The shard balancer does not take shard numbers into account when balancing shards across nodes. The reason shards 1 to 3 ended up on node 1 and shards 4 to 6 on node 2 is just due to the order in which shards were allocated to the nodes (which is somewhat deterministic). If you move shard 1 to node …

Thanks for the quick reply @ywelsch , I get what you're saying about the spread of the load, I agree. In my specific use case it does have some affect since I have a low stream of clients loading up a dashboard which executes a bunch of queries on different time spans with their own routing key, so t…

There is no way to tell ES where to place each shard on index creation (you would have to disable shard allocation altogether and manually allocate all shards - a big hassle). The best solution in this case is not to use routing but to just rely on term filters to select the right client. Term filt…

We've tested term filters but it was no match against routing, routing performed much better. As I understand it, routing also saves a lot of time in merging the filtered documents from different shards (is that accurate?), and since we have 30 shards for each index, that combined with relatively sm…

As a workaround you could try to create indices with a number of shards that's not a multiple of the number of nodes, they might then spread more evenly.

Discuss the Elastic Stack

Shard allocation and routing keys

Elastic Stack Elasticsearch

Topic		Replies	Views
Choosing which shard a document can go to? Elasticsearch	10	2399	July 5, 2017
Question about shard routing Elasticsearch	2	328	July 6, 2017
Re-balancing shard allocation Elasticsearch	21	940	June 20, 2018
Routing: Equal distribution on all shards Elasticsearch	5	697	June 28, 2019
Determine Shard Id based on routing key Elasticsearch	5	1222	July 6, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Shard allocation and routing keys

Related topics