When filebeat has little work to do, it stops loadbalancing correctly

We have 4 logstash instances running with the following configuration on all our servers:

output.logstash:
  enabled: true
  hosts: ["logstash1:5046", "logstash2:5046", "logstash3:5046", "logstash4:5046"] 
  loadbalance: true

On our busy servers I can see they maintain (and presumably use) all 4 connections. On our slow servers, (i.e. not much filebeat traffic), they drop down to just one connection. In the filebeat logs are errors like these:

2019-11-15T09:59:40.829-0800    ERROR   pipeline/output.go:121  Failed to publish events: write tcp 10.10.10.10:51422->10.10.10.11:5046: write: connection reset by peer

Which I understand is normal because logstash is resetting the connection due to inactivity. This is all fine. Filebeat settles down to using just a single logstash server.

The problem happens when that one logstash server that filebeat is using goes offline. Filebeat does not try any of the other servers, it just waits until that one servers comes back, (which may never happen).

filebeat version 6.8.3 (amd64), libbeat 6.8.3
Logstash version 6.8.3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.