We have 4 logstash instances running with the following configuration on all our servers:
output.logstash:
enabled: true
hosts: ["logstash1:5046", "logstash2:5046", "logstash3:5046", "logstash4:5046"]
loadbalance: true
On our busy servers I can see they maintain (and presumably use) all 4 connections. On our slow servers, (i.e. not much filebeat traffic), they drop down to just one connection. In the filebeat logs are errors like these:
2019-11-15T09:59:40.829-0800 ERROR pipeline/output.go:121 Failed to publish events: write tcp 10.10.10.10:51422->10.10.10.11:5046: write: connection reset by peer
Which I understand is normal because logstash is resetting the connection due to inactivity. This is all fine. Filebeat settles down to using just a single logstash server.
The problem happens when that one logstash server that filebeat is using goes offline. Filebeat does not try any of the other servers, it just waits until that one servers comes back, (which may never happen).
filebeat version 6.8.3 (amd64), libbeat 6.8.3
Logstash version 6.8.3