Lag is there from filebeat to Elasticsearch

Hi ,

I am using Filebeat in Kubernetes env. Data flow is like below
Filebeat -> Logstash -> Elasticsearch -> Kibana

I am using 7.9.0 version of filebeat

I am finding lag in FIlebeat to Kibana. ES & Logstash & Filebeat health looks fine. Could you help me here

Filebeat config:

filebeat.autodiscover:
     providers:
       - type: kubernetes
         scope: cluster
         node: ${NODE_NAME}
         hints.enabled: true
         add_resource_metadata:
           labels:
             enabled: true
         templates:
           - condition:
               equals:
                 kubernetes.labels.*****: "true"
             config:
               - type: container
                 multiline.pattern: '**Pattern***'
                 multiline.negate: false
                 multiline.match: after
                 paths:
                   - /var/log/containers/*-${data.kubernetes.container.id}.log
                 exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
                 fields:
                    cluster: {{ .Values.elk.clustername }}
                    container: ${data.kubernetes.container}

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}
    monitoring:
      enabled: true
      elasticsearch:
        hosts: ['https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
        username: ${ELASTICSEARCH_USERNAME}
        password: ${ELASTICSEARCH_PASSWORD}
        ssl:
          enabled: true
          verification_mode: none
    output.logstash:
      hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT}']

You've not provided any evidence of these delays, so it's a bit hard to help with what you have provided unfortunately.

@warkolm Please share what details do you need to take this forward. Will share it with you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.