Hi,
I wonder if there is a feature for dynamically changing the replication_factor of each index.
We have an Elasticseach cluster with hundreds of indices as part of our recommendations engine. Each index represents a different customer (encapsulation and privacy reasons).
Some of the indices (aka customers) are queried 300 times in a second while others can be queried once in a minute.
We are happy users of the "Adaptive Replica Selection" that help us to improve performance for those "heavy" (in term of search per second) indices.
Today we are manually set the replication_factor per each index: "heavy" indices get high replication_factor (5-6) while lighter get 1-2 - but that's not a scalable solution and required manual intervention.
As we grow fast, the idea is the keep the optimal number of shards and stop adding data nodes.
I wonder if such a feature exists? is it is on the roadmap, or is it something I can develop as a plugin?
The search rate can be retrieved from the index status and can assist to decide whether this index needs more or fewer replicas.
Thoughts?
Thanks,
Itay