Elastic stack 7.2.0 Filebeat -> Elasticsearch -> Kibana
I'm struggling with a weird scenario where I need to load the index template for my integration through Filebeat but I don't want to override the Filebeat template for
So far, my config looks like this:
# /etc/filebeat/filebeat.yml configuration file filebeat.modules: - module: mymodule alerts: enabled: true archives: enabled: false setup.template.json.enabled: true setup.template.json.path: '/etc/filebeat/my-template.json' setup.template.json.name: 'my-template' setup.template.overwrite: true output.elasticsearch.hosts: ['http://172.16.1.2:9200']
Since I'm using
setup.template.overwrite: true, it's overriding the Filebeat built-in template. Not a problem for my integration but the real problem comes when I activate the X-Pack monitoring features.
Filebeat monitoring events are being mapped with
my-template.json and that's something I don't want obviously.
Then once I open the
Monitoring section in Kibana it shows errors related to mapping of course.
My already researched solutions
- Disable monitoring for Filebeat, it works but then I'm forbidding users to see Filebeat stats.
- Launch the template manually before starting Filebeat service. Something I want to avoid because is an extra step for the user and it may fail if the order of the steps is not correct.
filebeat setup --index-management -E output.logstash.enabled=false -E setup.template.json.enabled=false -E 'output.elasticsearch.hosts=["172.16.1.2:9200"]'
Some questions about this
- Since we have a custom module for Filebeat, is there a way to set up our template just through the module? not the first time I asked this, but I got no clear response about it.
- Can I load two templates for Filebeat? I think no but let you answer this.
- Which is the way other users use to do this?
- Which is the way you guys recommend to manage this situation?
- Our indices are
- Our module just sets the index name and an ingest pipeline for Elasticsearch
- Our filebeat.yml configuration is using
setup.template.*settings to load template for our
- Also, once you activate the X-Pack monitoring features in 7.2, Filebeat starts to sends data to
filebeat*has no template because we are overriding it, then the Monitoring app in Kibana fails due to wrong mapping in
So the question is, how can we keep safe the Filebeat native template but also tell Filebeat that inserts ours?
Thanks in advance.