Filebeat Loadbalancing not working

I have setup a Logstash Cluster in Google Cloud that sits behind a Load Balancer and uses Autoscaling (-> when the load gets to high new instances are started up automatically).

Unfortunately this does not work properly with Filebeat. Filebeat only hits those Logstash Vms that existed when I started up Filebeat.

Example:
Lets assume I initially have those 3 Logstash hosts running:
Host1
Host2
Host3

When I startup Filebeat, it correctly distributes the messages to Host1, Host2 and Host3.

Now the Autoscaling kicks and and spins up 2 more instances, Host4 and Host5.

Unfortunately Filebeat still only sends messages to Host1, Host2 and Host3. The new hosts, Host4 and Host5, are ignored.

When I now restart Filebeat it sends messages to all 5 hosts!

So it seems Filebeat only sends messages to those hosts that have been running when Filebeat starts up.

My filebeat.yml looks like this:

filebeat.inputs:
- type: log
  paths:
  ...
  ...
  
output.logstash:
  hosts: ["logstash-loadbalancer:5044", "logstash-loadbalancer:5044"]
  worker: 1
  ttl: 2s
  loadbalance: true

I have added the same host (the loadbalancer) twice because I've read in the forums that otherwise Filebeat won't loadbalance messages -> I can confirm that.

But still loadbalancing seems to not work properly, e.g. TTL seems not to be respected because it always targets the same connections.

Looks like this was answered on GH in:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.