Filbeat to logstash extremely slow

Elastic Stack Beats
,
1

Hi everybody;

I have Filebeat 7.12 deployed as a DaemonSet on a EKS Cluster, fetching the logs from containers filtered by namespace; persistence queue on disk is enabled with a max size of 10GB.
Filebeat ships logs to a fleet of pods running logstash (in a different cluster), which ships them to AWS ElasticSearch.

I have 4 nodes running filebeat and everything is working fine, expect for one node, which is extremely slow sending logs (a delay close to 24 hours).

Here my configurations:

filebeat

filebeat:
  spool_size : 8192
  publish_async: true

filebeat.inputs:
  - type: container
    enabled: true
    paths:
      - /var/lib/docker/containers/**/*.log

    close_removed: true
    fields_under_root: true
    fields:
      index_prefix: k8s-logs
      
    processors:
      - decode_json_fields:
          fields: ["message"]
          max_depth: 2
          target: "msg"
          process_array: true
          overwrite_keys: true
      - add_kubernetes_metadata:    
      - add_fields:
          target: ''
          fields:
            source: "pod" 
      - drop_fields:
            fields: ["kubernetes.node"]
            ignore_missing: true              
      - drop_event.when:
          and:
            - not:
                equals:
                  kubernetes.namespace: namespace1
            - not:
                equals:
                  kubernetes.namespace: namespace2      


# Send events to Logstash.
output.logstash:
  enabled: true
  hosts: ["logstash-elb:5044"]
  bulk_max_size : 8192
  loadbalance: true
  workers: 4  
  timeout: 60
  template.name: filebeat
  template.path: filebeat.template.json

logging.level: debug
logging.to_files: false

queue.disk:
  max_size: 10GB

Logstash

input {
    beats {
       port => 5044
    }
    http { port => 8080 }
}

filter {
       
    if [rds]{
        mutate {
            add_field => { "source" => "rds" }
            }
    } else if [elb]{
        mutate {
            add_field => { "source" => "elb" }
            }
    }        
    if ![metadata]{
        if [source] != "sidecar" {
            if [kubernetes][namespace] {
                mutate {
                    add_field => { "namespace" => "%{[kubernetes][namespace]}" }
                }
            }
        }
    }
}

output {
    
    amazon_es {
        hosts => ["https://elasticsearch-aws:443"]
        ssl => true
        aws_access_key_id => "${AWS_ACCESS_KEY_ID}"
        aws_secret_access_key => "${AWS_SECRET_ACCESS_KEY}"
        region => "eu-central-1"
        index =>  "%{[namespace]}-logstash-%{+YYYY.MM.dd}"

    }

     s3{
         access_key_id  => "${AWS_ACCESS_KEY_ID}"
         secret_access_key => "${AWS_SECRET_ACCESS_KEY}"
         region => "eu-central-1"
         bucket => "mybucket-s3" 
         prefix => "logstash-logs-backup/%{[namespace]}"
     }
}

What am I missing? I can't figure out where is the bottleneck.
Thanks!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.