Separate indexes or combine?


#1

I want to store events in elasticsearch. All of the events have about 10 fields in common (like where it occurred, what severity, who reported it, etc) plus 2-6 fields of its own. For example event type A could have 3 fields of its own and event type B could have 6 fields of its own. There are 5-10 event types in total.

Originally I put them in their own separate indexes, like event_type_A and event_type_B, but anytime I needed to update one of the common field mappings (such as who reported the event), I had to update it across 5-10 different indexes and reindex each of them.

I thought it could be better to combine all the events into one index. If I need to add a new event type, is it possible to update the index mapping to add the new event's fields, or would I still need to reindex? I'm using dynamic: strict mappings because I want to define all fields up front.

Are there any disadvantages you can think of to combining all the event types into one index that has many more fields, as opposed to many indexes with fewer fields?

Would the unused fields in a given document take up any storage space? For example if event type A has 5 fields and there are a million of event type B in the index, would each of those type B documents have wasted space because of the 5 unused type A fields?


(Junaid) #2

Hi,

You can use PUT mappings in order to add new fields to your index. (https://www.elastic.co/guide/en/elasticsearch/reference/6.4/indices-put-mapping.html) This is expected to work with strict dynamic mappings.

I can think of one disadvantage for now. Lucene compression capability might be impacted when we have sparse data. Other than that, there shouldn't be an issue if you are combining different event_types. You'll have to consider your data volumes per event type. If it is expected to have few events for an event type, than it is better to aggregate them in a single index.