In our ES 6.8, we're planning to rethink our current indices partition as we acknowledged, month after month, that the starting decisions about indices was not properly planned.
We now got 3 TB of data (6 TB with replicas) in 145 indices, but there’s one of them that stores 99.9% of all data alone. Since we’re reworking them, we could use some advice.
We got a cluster of 3 identical nodes (also, a poor choice, but we will ask for other suggestions maybe later) with 16 CPUs, 64 GB RAM and disks with 2TB each. We’re planning to create a certain number of indices to spread the load of the single one, but we’re not sure about how many indices can we use in order to prevent a drop in performance.
Can you suggest the proper order of magnitude (10, 100, 1000, more?) regarding that number?
Here is more information:
- Our indices have more or less 100 fields (mappings) per document.
- Our single “super-index” is using 40 shards (this should be properly set according to the guidelines) while the other 144 indices have the default (5 for this version of ES).
- All indices are read and write.
Thank you for your help.