Filebeat events are not distributed equally to the logstash pods

Elastic Stack Logstash
1

I am using Filebeat, Logstash, Elastic search, Kibana in my Kubernetes cluster to monitor all my application logs.

Let me explain the issue and share my configurations.

  1. Filebeat collecting the logs and sending the events to the logstash pods, But I cloud observes load is not evenly distributed to the pods, tried with k8s service Cluster IP and Headless service.

  2. Due to uneven the load my logstash queues are failing up and not processing as fast as input events from the filebeat.

Filebeat config:

output.logstash: hosts: ['logstash-headless.logging.svc:5044'] loadbalance: true

Logstash Config:

queue.type: persisted path.queue: /usr/share/logstash/queue queue.max_bytes: 8gb queue.page_capacity: 512mb pipeline.batch.size: 9000 pipeline.batch.delay: 7

2

Hello and welcome,

In your case the configuration needs to be a little diferent.

The loadbalance option in the output should be used when beats are responsible for the load balancing, which means, when you have multiple hosts configured in the output, in your case you have only one hosts and the load balancing is done externally, not by beats.

Also, the connection from beats to Logstash is sticky, this can lead to uneven distribution when the destination host is a load balancer, which is your case, to avoid this you need to specify a ttl.

So, your output would need to be something like this:

output.logstash:
  hosts: ["logstash-headless.logging.svc:5044"]
  loadbalance: false
  pipelining: 0
  ttl: 5m

The pipelining is a setting related to async batches in Logstash, but since this does not work when ttl is used, you need to set it to 0.

You can change the value of the ttl if you want.

The documentation for these settings can be found here.

3

Thank you! @leandrojmp For quick response. Will follow your suggestions.