We have an ES 5x setup and we create around 39 indices on monthly basis.
Index distribution in the setup - https://gist.github.com/jay-dihenkar/80c9d3774b05027bd0f5db37df57b555
Cluster details -
Nodes = 3
Heap = 31g
We're keeping 15 months of data open in ES for analytics purpose.
Now, with ES 6x, we cannot have multiple types in a single index.
So on an average, if we break down these indices into a TYPE of its own, the count comes to around 80 indexes per month from 39.
This will be an ideal way for data modeling.
We can make a hack to add a custom type field say
custom_doc_type and keep indexing the same docs in the existing index. But ultimately it'll be a violation of data modeling rule and sparse index will be created.
Our dilemma is what's the ideal way to approach this problem, so we can keep the number of shards and open file in check while at the same time adhere to data modeling guidelines?
This data is only queried via Kibana and used for analytics purpose.