Wrong Mapping of ECS fields on fleet-managed datastreams causing multiple issues


We recently migrated part of our environment from beats to the elastic-agent managed by fleet integrations. While this makes agent management very easy, it introduced a lot of mapping issues which then cause search filters and SIEM rules to fail due to mapping inconsistencies accross datastreams, expecially when the starting index is logs-*.

Let's take a field like source.ip an example:
running GET logs-*/_mapping/field/source.ip shows mixed mappings


looking at a dataset system.system, the index template is composed of the following component templates

with its mappings coming from logs-system.system@package, which looks like this:

There is no other component templates with mapping for other ECS fields, which would explain the inconsistency.

Many other ECS fields have the same issue due to lack of mapping on the index template.
GET logs-*/_mapping/field/dns.resolved_ip


With these beeing fleet managed, when we try adding the mapping ourselves, it works but gets overwritten when an elastic update is applied to the integration.

Most of our SIEM rules querying these inconsistent fields just don't work :frowning_face:.

It would be useful to have an ecs mapping component template applied to all fleet managed datastreams.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.