We are on ES 5.2.2, just recently upgraded from ES 2.3, and are looking to reduce our shard count. We have been told numerous times by people involved with ES that our shard count seems very high. However, when we try to decrease it we run into horrible search performance and always end up back at our original high number.
For example we have one index(our indexes are split up by users) that is 385GB and we have 40 shards allocated for it. We tried to cut that number in half to 20 but found that search times doubled and were at a level that was too slow for users.
Here are our mappings. Our parent to child ratio is around 225/1. To do the search testing we ran 10 searches in a row with aggregations and benchmarked the overall time it took to return the results.
Any thoughts as to why we are unable to lower our shard counts without taking a big hit to search performance?