Filebeat connection reset by peer - new install

Elastic Stack Beats
1

ELK stack versions

Logstash: 5.1.2
Kibana: 5.1.2
Elasticsearch:5.1.2

Logstash stack successfully receives syslog events on port 5514, but any filebeat logs send to port 5044 are failing. This has never worked.

/var/log/filebeat

2017-02-01T10:14:03Z ERR Failed to publish events caused by: read tcp 192.168.16.246:49754->192.168.16.247:5044: read: connection reset by peer
2017-02-01T10:14:03Z INFO Error publishing events (retrying): read tcp 192.168.16.246:49754->192.168.16.247:5044: read: connection reset by peer
2017-02-01T10:14:25Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=401 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1 libbeat.logstash.published_but_not_acked_events=805
2017-02-01T10:14:55Z INFO No non-zero metrics in the last 30s

/etc/filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/syslog
        - /var/log/dpkg.log
        - /var/log/unattended-upgrades/unattended-upgrades.log

      input_type: log
      
      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["192.168.16.247:5044"]
    bulk_max_size: 1024

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

/etc/logstash/02-beats-input.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    type => "file-beats"
    tags => "file-beats"
  }
}
2

which logstas-input-beats version is installed? Logstash 5.2 has been released recently, it might have a newer plugin version.

3

logstash-input-beats (3.1.12)

Updating all the plugins and restarting logstash I cannot see some events, but the original host that I was configuring still have the same problem. Although I do know have two other hosts successfully sending beat events.

4

So you still get connection reset by peer after updating the plugin to version 3.1.12?

You have some logstash debug logs?

5

Hello!

I'm starting with the elastic stack and I'm suffering the same problems.... I had tried in Ubuntu and Mac with same results...

The event propagation between filebeats and elastic search works perfectly fine... the problem appears when you try to use log stash... for some reason, using the latests versions of all the elements, log stash just simply rejects filberts messages.

I read that it could be a bug related with the timeout in other post but I already tried that without success...

I'm using elastic search, log stash, filebeat 5.1.2 with Java 8 from oracle

Any ideas?

6

@caleneo Can you also share your Logstash and Filebeat configs and logs?

7

Hi Steffens,

It actually now works for one machine, with the exact same configs ( as I'm deploying via puppet) but the first machine 192.168.16.246 still has the errors.

Im starting to think the servers in questions has a problem with the certificate, as before puppet automation I installed a older version on this server.

[2017-02-02T10:37:11,155][ERROR][org.logstash.beats.BeatsHandler] Exception: not an SSL/TLS record: 325700000001324300000187785e6cd1b18ed34010c6f1c06bd08ca602c917db9b38176f751505a23b2a0842e3cdd819b1d95ded8e13ace8de1d1d3a68b8f693bee6f7379f56abd59bd5eaedbb1b3e9c59e9484a686fa84b62b45896e2e384150e4c8a1647f13c30293e5578e6526862b4f8910780169a9d359ded5a48734aac893cd473c9f520a1fe7bfc66ba6dffdd82660a25c5aceb29dad674161e1f3f83f3c2416124f17c048de06208ec14aea2270bbfbaa6b7e038ab8ce248198a4c818f302c30879f215e03d0aca7984517789f622932f805067634178638c2015d5e92c63a17b270e12ca33852890138e7980f08d7937806cd8b840934c285b38c0b380a47399232d0aca7984517709c554671a40c072c9c85bc6deffbb6d96c3766bfe99b7ebfdf986e77c00f58e11f397bc340e767dc34bf306185a758f495f9c2b9480c68b15bb76bf3ccfea072e6a2744e68d134edfd5d63ee9af64bd3db7663bb6e6d76dbaf58618973768c16eb0be5dac7a9fe17338e6361456bfaa6faafb48434eb8f97fe3e4ef8f43b0000ffff0e4bb343
8

hm... this error message looks like logstash is expecting the TLS handshake, but filebeat is configured to use plaintext. Check the config file for the machine still producing the error.

9

What I found that was the machine failing had two configuration files, so I deleted the incorrect one and restart filbeat. Same problem. I thought id better also restart logstash which I have now done so letting it run for a few minutes to see what happens.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.