Logstash is not accepting my Filebeat logs

Elastic Stack Logstash
#1

Filebeat is on a prod server and Logstash is on a different servers.. I can not figure out why Logstash is rejecting my Filebeat log pushes. Below is the Logstash error. And below that shows telnet to port 5044 works. ELK versions aree 5.2.2

2017-03-06T17:47:24Z INFO Error publishing events (retrying): read tcp 10.0.0.165:39710->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:47:39Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=4 libbeat.logstash.publish.read_errors=5 filebeat.harvester.started=4 filebeat.harvester.running=4 libbeat.publisher.published_events=2044 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.published_but_not_acked_events=10220 libbeat.logstash.publish.write_bytes=2180
2017-03-06T17:47:40Z ERR Failed to publish events caused by: read tcp 10.0.0.165:39716->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:47:40Z INFO Error publishing events (retrying): read tcp 10.0.0.165:39716->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:48:09Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=308 libbeat.logstash.published_but_not_acked_events=2044 libbeat.logstash.call_count.PublishEvents=1
2017-03-06T17:48:12Z ERR Failed to publish events caused by: read tcp 10.0.0.165:39774->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:48:12Z INFO Error publishing events (retrying): read tcp 10.0.0.165:39774->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:48:39Z INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=307 libbeat.logstash.published_but_not_acked_events=2044
2017-03-06T17:49:09Z INFO No non-zero metrics in the last 30s
2017-03-06T17:49:12Z ERR Failed to publish events caused by: read tcp 10.0.0.165:39788->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:49:12Z INFO Error publishing events (retrying): read tcp 10.0.0.165:39788->10.0.0.212:5044: read: connection reset by peer
2017-03-06T17:49:39Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=311 libbeat.logstash.published_but_not_acked_events=2044 libbeat.logstash.publish.read_errors=1 libbeat.logstash.call_count.PublishEvents=1
2017-03-06T17:50:09Z INFO No non-zero metrics in the last 30s

ubuntu@ip-10-0-0-165:/etc/filebeat-PROD!!!$ telnet 10.0.0.212 5044
Trying 10.0.0.212...
Connected to 10.0.0.212.

#2

Have you configured Logstash but not Filebeat to use SSL, or vice versa?

#3

Here is the logstash config file for input on the ELK server

cat 02-beats-input.conf
input {
beats {
port => 5044
type => "logs"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
client_inactivity_timeout => "36000"
}
}

And here is the filebeat.yml from the prod server. Please ignore the comments - i've been trying everything i read on the forum to get to work.

cat /etc/filebeat/filebeat.yml
filebeat:
prospectors:
-
paths:
#- /var/log/.log
- /var/log/syslog
- /var/log/apache2/.log # NEW!
input_type: log
registry_file: /var/lib/filebeat/registry

output.logstash:

The Logstash hosts

#hosts: ["localhost:5044"]

hosts: ["10.0.0.212:5044"]

#hosts: ["elkserver.planetecosystems.com:5044"]

bulk_max_size: 1024

tls:

# List of root certificates for HTTPS server verifications

certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

output:
logstash:
enabled: true
hosts: ["10.0.0.212:5044"]
worker: 1
tls:
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
shipper:
logging:
files:

------ Telnet works from prod box

telnet 10.0.0.212 5044
Trying 10.0.0.212...
Connected to 10.0.0.212.
Escape character is '^]'.

#4

Please format your filebeat.yml as preformatted text when posting it here. If you don't all indentation is lost and indentation matters with YAML files.

Have you tried disabling SSL on the Logstash side? If that works you know that the problem is that Filebeat doesn't use SSL.

#5

ok good call. I changed Logstash to ssl => false
and now i see f filebeat log index in: curl 'localhost:9200/_cat/indices?v'

thanks. so now how do i figure out why ssl is failing? If the public cert was bad then i assume the error would state that. What else so i explore to get ssl working

#6

I repeat: Please format your filebeat.yml as preformatted text when posting it here. If you don't all indentation is lost and indentation matters with YAML files.

#7

Sorry, thought that was going forward.

filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        #- /var/log/*.log
        - /var/log/syslog
        - /var/log/apache2/*.log # NEW!
      input_type: log
  registry_file: /var/lib/filebeat/registry
  #output.logstash:
  #  # The Logstash hosts
  #  #hosts: ["localhost:5044"]
  #  hosts: ["10.0.0.212:5044"]
  #  #hosts: ["elkserver.planetecosystems.com:5044"]
  #  bulk_max_size: 1024
  #  tls:
  #    # List of root certificates for HTTPS server verifications
  #    certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
output:
  logstash:
    enabled: true
    hosts: ["10.0.0.212:5044"]
    worker: 1
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
shipper:
logging:
  files:
#8

What version of Filebeat is this? According to https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html the ssl key enables SSL. There's no mention of a tls key.

#9

Version: 5.2.2

#10

ok cool. i changed the yml to use ssl and changed logstash back to ssl => true
looks good now. thanks

#11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.