I am using ECK1.6 to deploy Elasticsearch cluster 7.15.2 on k8s,and then using ECK to deploy filebeat to collect Elasticsearch server log, gc log, slowlog, deprecation log. There is an error as same as this issue Filebeat registry collisions when an integration is used twice #22054 when I using autodiscover feature and configure a template for echo types of log (server log, gc log, slowlog, deprecation log). While everything is OK when I comment out other three template and collect server logs only.
So, Is there is method to collect multiple types of Elasticsearch logs when filebeat and Elasticsearch are deployed with ECK?
My filebeat manifest is modified from filebeat_autodiscover_by_metadata.yaml
Not sure if it helps, but worth trying to add IDs:
- condition.equals.kubernetes.namespace: log-namespace
config:
- paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
type: container
id: "first"
- condition.equals.kubernetes.labels.log-label: "true"
config:
- paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
type: container
id: "second"
Thank you.
The same error is returned with "id" set. My filebeat.yaml is like this below:
- condition
equals:
kubernetes:
namespace: elastic-system
labels:
common_k8s_elastic_co/type: elasticsearch
config:
- module: elasticsearch
slowlog:
input:
id: "slowlog"
paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
type: container
include_lines:
- "type":\s"(index_search_slowlog|index_indexing_slowlog)"
- condition
equals:
kubernetes:
namespace: elastic-system
labels:
common_k8s_elastic_co/type: elasticsearch
config:
- module: elasticsearch
server:
input:
id: "server"
paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
type: container
include_lines:
- "type":\s"server"}
I find a perfect solution, it helped.
The solution is from this post:
https://xeraa.net/blog/2020_filebeat-modules-with-docker-kubernetes/#planD
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.