Hi,
i have 55 client nodes in my Elastic cluster, which are all configured as back-end servers under HAProxy.
In Logstash, at output filter, the 'hosts =>' configuration is set to HAProxy address, and for weeks everything works without an issue.
few days ago one of the clients of ES died, and HAProxy skipped it correctly.
however, my Logstash instance insisted on attempting to reconnect, filling the log file with thousands of lines with the following error: "Attempted to resurrect connection to dead ES instance".
i am certain that HAProxy did not return the client that went down.
So, i am wondering:
- Does logstash connect once to a client and keeps this connection alive as long as Logstash is running, regardless of how many clients there are in the cluster?
- Why doesnt Logstash give up on this host? there are thousands of attempts in the log file, spanning over 4 days, and plenty of other clients that are available to serve it
thanks,
Ofer