INFO Error publishing events (retrying): read tcp read: connection reset by peer

I put filebeat on a prod box and can not get filebeat to push to Logstash on my elkserver box. I am getting the below error from the filebeat.log file

INFO Error publishing events (retrying): read tcp IP:45790->ELK_IP:5044: read: connection reset by peer

telnet works for that ip:5044. I tried making the ip addresses match on filebeat.yml and Logstash config but does not work. Logstash only seems to work if host is localhost. I see nothing in the logs telling me what the issue is. Telnet to port 5044 works so I am at a loss to understand why filebeat can not push a log.

filebeat.yml from Prod box:
...
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

#hosts: ["localhost:5044"]
hosts: ["xx.0.0.xxx:5044"]
tls:
# List of root certificates for HTTPS server verifications
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================
...

Logstash config from my elserver box

cat /etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

cat /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["xx.0.0.xxx:9200"]
#hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

I think it is something between the yml and output config but can not figure out how to get it working

I checked my ssl cert using: curl -v --cacert /etc/pki/tls/certs/logstash-forwarder.crt https://xx.0.0.xxx:5044
and i think the below means it is ok -

ALPN, offering http/1.1

  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  • server certificate verification OK
  • server certificate status verification SKIPPED
  • error fetching CN from cert:The requested data were not available.
  • common name: (matched)
  • server certificate expiration date OK
  • server certificate activation date OK
  • certificate public key: RSA
  • certificate version: #3
  • subject: C=AU,ST=Some-State,O=Internet Widgits Pty Ltd
  • start date: Fri, 03 Mar 2017 21:11:40 GMT
  • expire date: Mon, 01 Mar 2027 21:11:40 GMT
  • issuer: C=AU,ST=Some-State,O=Internet Widgits Pty Ltd
  • compression: NULL
  • ALPN, server did not agree to a protocol

GET / HTTP/1.1
Host: 10.0.0.212:5044
User-Agent: curl/7.47.0
Accept: /

  • Empty reply from server
  • Connection #0 to host 10.0.0.212 left intact
    curl: (52) Empty reply from server

I added a 10 minute timeout to Logstash config and still get the below filebeat error log

2017-03-04T17:18:52Z ERR Failed to publish events caused by: read tcp 10.0.0.xxx:34916->10.0.0.xxx:5044: read: connection reset by peer
2017-03-04T17:18:52Z INFO Error publishing events (retrying): read tcp 10.0.0.xxx:34916->10.0.0.xxx:5044: read: connection reset by peer
2017-03-04T17:19:09Z INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=381 libbeat.logstash.published_but_not_acked_events=2044

Here are my versions loaded on the elkserver

curl -XGET 'localhost:9200'
{
"name" : "elkserver-node-1",
"cluster_name" : "elkserver-cluster-1",
"cluster_uuid" : "mxhWhcThS3uioz53lkj4FA",
"version" : {
"number" : "5.2.2",
"build_hash" : "f9d9b74",
"build_date" : "2017-02-24T17:26:45.835Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.