ILM breaks the mapping for a filebeat module

Elastic Stack Beats
1

I installed a filebeat with google cloud module (https://github.com/elastic/beats/tree/master/x-pack/filebeat/module/googlecloud) and everything works fine, data gets parsed, and it shows up in the SIEM app just fine (ECE). After I had added ILM to the index, things just broke, and I think it's the mapping related issue.

What's the best way to fix it? Right now, I added a custom mapping to a template under ILM, but I really think there might be a better way of doing it.

2

I worked through this issue separately with Daniel.

The scenario was the following:

  • Filebeat ingesting from a set of sources, to filebeat-7.x.x-*
  • Filebeat ingesting from another set of sources, but want to send to a different index which adds additional fields, has different shard settings, and different ILM policy
  • new index template was set up for the new set of sources, some field mapping definitions were changed, and data wasn't working with SIEM due to the changes
  • questions around what happens when the filebeat mappings are expanded in future releases, i.e. how would they keep the new index template in sync

The solution was the following:

  • leverage the fact that index templates are inheritable, e.g. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
  • create index template e.g. filebeat-7.x.x-additional-* for the additional sources would inherit everything from filebeat-7.x.x-* pattern since the glob pattern would cover both, and in the new index template, override the shard settings and ILM settings. new field mappings would be added to the new index template.
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.