ES for logs -- field conflict hell 🔥

Elastic Stack Elasticsearch
17

Oh, also,

  • conserve fields
    • consider using multiple indices / streams to cope with large quantities of fields

For example, if you find your logs are eating up over a thousand fields, perhaps split them across multiple series/streams:

  • logs-all_logs-a.2024-03-29 or logs-datastream-all_logs-a gets 1000 fields
  • logs-all_logs-b.2024-03-29 or logs-datastream-all_logs-b gets the overage

Then query across both sets, a and b, using a single data view / index pattern.

I think this should mitigate the "too many fields" problem as mentioned in the mapping limit settings doc, but I'm not entirely sure and would love to have an elastician chime in.

[edit: You would also want to minimize as much as possible field conflicts between the series to avoid Kibana balking.]

[edit: Oh, apparently this has already been suggested by @warkolm! Thanks, Mark!]

1 Like