How to scale Filebeat output in sync mode

Elastic Stack Beats
1

Hey everyone,

I'm using Filebeat in sync mode (pipelining disabled) and TTL set, as I have a simple loadbalancer as a host.

If I use this configuration on a host that has a high log rate, I end up with a bottleneck on Filebeat, which cannot send the logs fast enough to the Logstash. Looks like the worker's settings in sync mode do not make any difference, as there is no other host to make more connections to. Changing the size of the batch (8192) will result in timeouts, as Logstash is not able to accept it.

Is there any recommended way how to scale out the Filebeat output to the Logstash in this mode?

Filebeat config:

max_bytes: 262144
fields_under_root: true
fields:
  cluster_name: example

logging:
  json: true
  metrics.period: 5m
  level: INFO
processors:
  - add_cloud_metadata:

filebeat.autodiscover:
  providers:
    - type: Kubernetes
      hints.enabled: true
      hints.default_config:
        type: container
        close_removed: false
        paths:
          - /var/log/containers/*-${data.container.id}.log

queue.mem:
  events: 32000
  flush.min_events: 8192
  flush.timeout: 0s

http:
  enabled: true
  host: localhost
  port: 5066

output:
  logstash:
    hosts: ["logstash-loadbalancer.example.:5045"]
    bulk_max_size: 4096
    worker: 2
    pipelining: 0
    ttl: 90s
2

How did you track the bottleneck to be in Filebeat and nothing in your Logstash output?

If the bottleneck is in the Logstash output it will tell filebeat to backoff.

Also, what do you have in both Filebeat and Logstash logs? If there is some backing off happening you should have some information in the logs.

Do you know what is the rate?

3

Hey @leandrojmp,

I track the Filebeat bottleneck with the Filebeat exporter. It shows that the Filebeat memory queue is constantly full, the Filebeat active events grow up over time, and Logstash is almost idle. When I switch to async mode, scaling out using the Filebeat workers, Logstash can deliver x-times more than with the sync setup.

There are no warnings or errors in Logstash logs.

The high log rate is around 3K-4K lines per second (each line <1MB)