Connection reset by peer with loadbalance: true filebeat => logstash

Elastic Stack Beats
1

Hello, i have been looking into the issue between filebeat and logstash for days now and have unfortunately gotten nowhere. Namely the configuration we have is as following:

  • Test Environment (firewall and SELinux disabled)
  • Everything is running on Centos7 (1 Server with filebeat 2x Server with logstash sending to an Elasticsearch cluster with 2 nodes, one being a master node)
  • Pipeline as follows Filebeat => Logstash => Elasticsearch

filebeat.yml configuration

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["10.226.95.10:5044","10.226.95.13:5045"]
loadbalance: true

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

logstash.conf file

input {
beats {
port => 5044
client_inactivity_timeout => 86400
}

}

output {

elasticsearch {

hosts => ["http://10.226.95.10:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]-%{+YYYY.MM.dd}"

}

once filebeat is started with stdout (filebeat -e) dashboards and templates have been loaded in kibana, the following is message is received:

Connecting to backoff(async(tcp://10.226.95.10:5044))
Connecting to backoff(async(tcp://10.226.95.13:5045))
Connection to backoff(async(tcp://10.226.95.10:5044)) established
Failed to publish events caused by: read tcp 10.226.95.12:53112->10.226.95.10:5044: read: connection reset by peer
Failed to publish events caused by: read tcp 10.226.95.12:53112->10.226.95.10:5044: read: connection reset by peer
Failed to publish events caused by: client is not connected
Failed to connect to backoff(async(tcp://10.226.95.13:5045)): dial tcp 10.226.95.13:5045: connect: connection refused
Attempting to reconnect to backoff(async(tcp://10.226.95.13:5045)) with 1 reconnect attempt(s)
Failed to publish events: client is not connected
Connecting to backoff(async(tcp://10.226.95.10:5044))
Failed to connect to backoff(async(tcp://10.226.95.13:5045)): dial tcp 10.226.95.13:5045: connect: connection refused
Attempting to reconnect to backoff(async(tcp://10.226.95.13:5045)) with 2 reconnect attempt(s)
Failed to connect to backoff(async(tcp://10.226.95.10:5044)): dial tcp 10.226.95.10:5044: connect: connection refused
Attempting to reconnect to backoff(async(tcp://10.226.95.10:5044)) with 1 reconnect attempt(s)

I have found that the loadbalance:true setting might be the most likely culprit, as once removed and only one logstash is set as an output filebeat can and sucessfully connects to either of the nodes set, but as soon as a second one is added and loadballance turned on the error returns.
Additionally i have attempted using the options for worker: and have set the timeout to from 60, 3000 and finally 86400 none made a difference.

Any help will be more than appreciated.

2

Connection refused error means that the TCP connection could not be established, because the 'endpoint' did actively return a packet saying: no. The error is not created by Beats, but comes from your network (or target hosts). Do you have any firewall rules (or other equipment), which would limit the number of connections? Is logstash running on these machines? Are the ports correct?

3

Hi Steffens, yes the ports are correct, logstash is running on both servers and i turned off firewall and SELinux on all the machines for testing purposes.
However we have deicided to place haproxy in front of logstash and have it handle load balancing and redundancy, so this can be closed. Thank you for your support

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.