Hi team,
Elastic stack 7.2.0 Filebeat -> Elasticsearch -> Kibana
I'm struggling with a weird scenario where I need to load the index template for my integration through Filebeat but I don't want to override the Filebeat template for filebeat*
indices.
So far, my config looks like this:
# /etc/filebeat/filebeat.yml configuration file
filebeat.modules:
- module: mymodule
alerts:
enabled: true
archives:
enabled: false
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/my-template.json'
setup.template.json.name: 'my-template'
setup.template.overwrite: true
output.elasticsearch.hosts: ['http://172.16.1.2:9200']
Description
Since I'm using setup.template.overwrite: true
, it's overriding the Filebeat built-in template. Not a problem for my integration but the real problem comes when I activate the X-Pack monitoring features.
Filebeat monitoring events are being mapped with my-template.json
and that's something I don't want obviously.
Then once I open the Monitoring
section in Kibana it shows errors related to mapping of course.
My already researched solutions
- Disable monitoring for Filebeat, it works but then I'm forbidding users to see Filebeat stats.
- Launch the template manually before starting Filebeat service. Something I want to avoid because is an extra step for the user and it may fail if the order of the steps is not correct.
filebeat setup --index-management -E output.logstash.enabled=false -E setup.template.json.enabled=false -E 'output.elasticsearch.hosts=["172.16.1.2:9200"]'
Some questions about this
- Since we have a custom module for Filebeat, is there a way to set up our template just through the module? not the first time I asked this, but I got no clear response about it.
- Can I load two templates for Filebeat? I think no but let you answer this.
- Which is the way other users use to do this?
- Which is the way you guys recommend to manage this situation?
Brief summary
- Our indices are
wazuh-alerts-3.x-*
- Our module just sets the index name and an ingest pipeline for Elasticsearch
- Our filebeat.yml configuration is using
setup.template.*
settings to load template for ourwazuh-alerts-3.x-*
indices - Also, once you activate the X-Pack monitoring features in 7.2, Filebeat starts to sends data to
filebeat*
indices - The
filebeat*
has no template because we are overriding it, then the Monitoring app in Kibana fails due to wrong mapping infilebeat*
indices
So the question is, how can we keep safe the Filebeat native template but also tell Filebeat that inserts ours?
Thanks in advance.
Best regards,
Jesús