Adjusting custom mapping for new fields

[daniel_a (daniel_aug)]

I'm looking for the best approaches to adjust an index mapping every time there are new fields added to it (after a vendor or a beat update for example).

Currently, the filebeat (7.6) googlecloud module (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-googlecloud.html) added a few new fields for flow logs. All new fields showed up as text and keyword since they were dynamically mapped and added to my custom mapping (in the custom mapping, all fields were changed from text to either keyword, ip, or geo type).

How would I fix this now and in the future, this seems like an ongoing issue?

[Jason_Slater]

I worked through this issue separately with Daniel.

The scenario was the following:

• Filebeat ingesting from a set of sources, to filebeat-7.x.x-*
• Filebeat ingesting from another set of sources, but want to send to a different index which adds additional fields, has different shard settings, and different ILM policy
• new index template was set up for the new set of sources, some field mapping definitions were changed, and data wasn't working with SIEM due to the changes
• questions around what happens when the filebeat mappings are expanded in future releases, i.e. how would they keep the new index template in sync

The solution was the following:

• leverage the fact that index templates are inheritable, e.g. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
• create index template e.g. filebeat-7.x.x-additional-* for the additional sources would inherit everything from filebeat-7.x.x-* pattern since the glob pattern would cover both, and in the new index template, override the shard settings and ILM settings. new field mappings would be added to the new index template.