We have approximately ~17TB of data in our cluster. I was looking at a 6 month sample of data index breakdowns for our cluster. I'm seeing 90% of the indexes are pretty small in size. They can range anywhere from 50KB up to 48MB in size. We also have 3 groups of indices which make up the most of the data. They can range from 11-14GB in size. For the smaller indices, they are time based broken up by hour. The group of larger indexes are also time based but broken up by days.
It sounds like recommendation is to decrease the number of shards in order to collapse the smaller indices into larger indices. Do you have a recommendation for doing that with version 2 of ES? Would we just need to reindex the data? Any thoughts are appreciated!