Hi
We operate a >100 nodes cluster, the size mainly intended to allow requests on a big temporal depth.
We are daily creating a few separate indices for indexing ( up to ten shard per indice + replicas ).
We notice that shards from these different indices are often allocated to the same node, leading to high resources contention.
Indices are created at the same hour ( midnight).
Any idea how we can have allocation to separate nodes ?
P.s. of course we have limited to 1 shard per node these indices...
If you don't want particular indices to be co-located then the simplest solution is to define the sets of nodes for each index up-front, using node attributes and shard allocation filtering.
It's not unusual to only want to apply this restriction while the index is actively writing documents. If so, you can remove the allocation filter once the next index becomes active.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.