Filebeat Loadbalancing not working

Elastic Stack Beats
1

I have setup a Logstash Cluster in Google Cloud that sits behind a Load Balancer and uses Autoscaling (-> when the load gets to high new instances are started up automatically).

Unfortunately this does not work properly with Filebeat. Filebeat only hits those Logstash Vms that existed when I started up Filebeat.

Example:
Lets assume I initially have those 3 Logstash hosts running:
Host1
Host2
Host3

When I startup Filebeat, it correctly distributes the messages to Host1, Host2 and Host3.

Now the Autoscaling kicks and and spins up 2 more instances, Host4 and Host5.

Unfortunately Filebeat still only sends messages to Host1, Host2 and Host3. The new hosts, Host4 and Host5, are ignored.

When I now restart Filebeat it sends messages to all 5 hosts!

So it seems Filebeat only sends messages to those hosts that have been running when Filebeat starts up.

My filebeat.yml looks like this:

filebeat.inputs:
- type: log
  paths:
  ...
  ...
  
output.logstash:
  hosts: ["logstash-loadbalancer:5044", "logstash-loadbalancer:5044"]
  worker: 1
  ttl: 2s
  loadbalance: true

I have added the same host (the loadbalancer) twice because I've read in the forums that otherwise Filebeat won't loadbalance messages -> I can confirm that.

But still loadbalancing seems to not work properly, e.g. TTL seems not to be respected because it always targets the same connections.

2

Looks like this was answered on GH in:

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.