Hi ,
I am using Filebeat in Kubernetes env. Data flow is like below
Filebeat -> Logstash -> Elasticsearch -> Kibana
I am using 7.9.0 version of filebeat
I am finding lag in FIlebeat to Kibana. ES & Logstash & Filebeat health looks fine. Could you help me here
Filebeat config:
filebeat.autodiscover:
providers:
- type: kubernetes
scope: cluster
node: ${NODE_NAME}
hints.enabled: true
add_resource_metadata:
labels:
enabled: true
templates:
- condition:
equals:
kubernetes.labels.*****: "true"
config:
- type: container
multiline.pattern: '**Pattern***'
multiline.negate: false
multiline.match: after
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
fields:
cluster: {{ .Values.elk.clustername }}
container: ${data.kubernetes.container}
processors:
- add_cloud_metadata:
- add_host_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
monitoring:
enabled: true
elasticsearch:
hosts: ['https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl:
enabled: true
verification_mode: none
output.logstash:
hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT}']