Respect the Filebeat template when using "setup.template", X-Pack monitoring failing

[jesusgn90]

Hi team,

Elastic stack 7.2.0 Filebeat -> Elasticsearch -> Kibana

I'm struggling with a weird scenario where I need to load the index template for my integration through Filebeat but I don't want to override the Filebeat template for filebeat* indices.

So far, my config looks like this:

# /etc/filebeat/filebeat.yml configuration file
filebeat.modules:
  - module: mymodule
    alerts:
      enabled: true
    archives:
      enabled: false

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/my-template.json'
setup.template.json.name: 'my-template'
setup.template.overwrite: true 

output.elasticsearch.hosts: ['http://172.16.1.2:9200']

Description

Since I'm using setup.template.overwrite: true, it's overriding the Filebeat built-in template. Not a problem for my integration but the real problem comes when I activate the X-Pack monitoring features.

Filebeat monitoring events are being mapped with my-template.json and that's something I don't want obviously.

Then once I open the Monitoring section in Kibana it shows errors related to mapping of course.

My already researched solutions

1. Disable monitoring for Filebeat, it works but then I'm forbidding users to see Filebeat stats.
2. Launch the template manually before starting Filebeat service. Something I want to avoid because is an extra step for the user and it may fail if the order of the steps is not correct.

filebeat setup --index-management -E output.logstash.enabled=false -E setup.template.json.enabled=false -E 'output.elasticsearch.hosts=["172.16.1.2:9200"]'

Some questions about this

1. Since we have a custom module for Filebeat, is there a way to set up our template just through the module? not the first time I asked this, but I got no clear response about it.
2. Can I load two templates for Filebeat? I think no but let you answer this.
3. Which is the way other users use to do this?
4. Which is the way you guys recommend to manage this situation?

Brief summary

• Our indices are wazuh-alerts-3.x-*
• Our module just sets the index name and an ingest pipeline for Elasticsearch
• Our filebeat.yml configuration is using setup.template.* settings to load template for our wazuh-alerts-3.x-* indices
• Also, once you activate the X-Pack monitoring features in 7.2, Filebeat starts to sends data to filebeat* indices
• The filebeat* has no template because we are overriding it, then the Monitoring app in Kibana fails due to wrong mapping in filebeat* indices

So the question is, how can we keep safe the Filebeat native template but also tell Filebeat that inserts ours?

Thanks in advance.

Best regards,
Jesús

[Mike_Place]

Instead of overwriting the template, could you use setup.template.append_fields just to append the fields that you need?

Obviously, this may still pollute the monitoring template but its functionality may be preserved. I'm still looking into other options as well.