I think this question is best described by using an example.
Say I have one index called "myindex". The fields in this index have the following structure:
- Each document in this index contains a field called "group" which has a value between 1 and 100.
- Each document in the index has 20 fields which have the same name and datatype across all documents.
- Each document in group x has 5-20 fields which are only used by group x.
Should all these documents be stored in one index or should I make one index per group?
Some of the searches and visualisations I use in kibana will be across many groups. This means that if I split the data into one index per group there will be searches and aggregations done over many elasticsearch-indecies. Will this impact performance?
If I keep everything in one index, it will contain hundreds of fields, but only 25-40 fields will be "used" in each document. How will this impact performance?
Are there any other ways of structuring this, other than the two options I have considered in this post?
Thanks.