Latest filebeat binary failing

[Sai_Birada]

Logs are flowing fine to logstash-2.4.1 if I am using filebeat-1.2.3 binary. However as part of upgradation when I am trying latest filebeat binary 5.1.2. Filebeat is failing with error

2016-12-26T03:42:46-08:00 DBG connect
2016-12-26T03:42:46-08:00 DBG Try to publish 2002 events to logstash with window size 1
2016-12-26T03:42:46-08:00 DBG handle error: EOF
2016-12-26T03:42:46-08:00 DBG closing
2016-12-26T03:42:46-08:00 DBG 0 events out of 2002 events sent to logstash. Continue sending
2016-12-26T03:42:46-08:00 DBG close connection
2016-12-26T03:42:46-08:00 ERR Failed to publish events caused by: EOF
2016-12-26T03:42:46-08:00 INFO Error publishing events (retrying): EOF
2016-12-26T03:42:46-08:00 DBG close connection
2016-12-26T03:42:46-08:00 DBG send fail

I looked at the version matching matrix and it says filebeat-5.1 will work with logstash-2.4.1. Error looks like a network issue, but if I use 1.2.3 binary its working good. Is there any configuration specifications required specially for the latest filebeat?.

[tudor (Tudor Golubenco)]

What logstash-input-beats plugin version do you have? Try to upgrade it, please.

[Sai_Birada]

I am using logstash-2.4.1 with recent 3.1.2 logstash-input-beats plugin. I am using 5.1.2 filebeat binary. Following is my logstash config.

input {
beats {
port => 443
ssl => true
ssl_certificate_authorities => ["/tmp/cacert_filebeat.pem"]
ssl_certificate => "/tmp/ls-zs.pb.zscaleranalytics.net.crt"
ssl_key => "/tmp/ls-zs.pb.zscaleranalytics.net.key.pem"
ssl_verify_mode => "force_peer"
}
}
filter {
}
output {
stdout { }
elasticsearch {
hosts => [ "10.10.35.66:9200" ]
index => "heartbeatlogs_%{+YYYY.MM.dd}"
}
}

Following is my filebeat config.

filebeat:
prospectors:
-
paths:
- /var/log/heartbeat.log
fields:
hostip: "10.10.35.180"
cloudname: "zscaler.net"
document_type: heartbeat_Etc/GMT+0
close_older: 5m

output:
logstash:
hosts: ["ls-zsn.pb.zscaleranalytics.net:443"]
loadbalance: true
tls:
certificate_authorities: ["/sc/filebeat/certs/cacert_logstash.pem"]
certificate: "/sc/filebeat/certs/fbt-zsn.crt"
certificate_key: "/sc/filebeat/certs/fbt-zsn.key"

logging:
to_syslog: false
to_files: true
level: debug
files:
path: /sc/filebeat
name: auditlog.log
rotateeverybytes: 10485760
keepfiles: 2

[Sai_Birada]

This is a config issue. Its working fine with the following config modification

output.logstash:
hosts: ["ls-zsn.pb.zscaleranalytics.net:443"]
ssl.certificate_authorities: ["/sc/filebeat/certs/cacert_logstash.pem"]
ssl.certificate: "/sc/filebeat/certs/fbt-zsn.crt"
ssl.key: "/sc/filebeat/certs/fbt-zsn.key"

Thanks.

[steffens (Steffen Siering)]

I'd recommend to update the logstash-input-beats plugin to the more recent version. The 3.x plugin version is a complete rewrite and there have been some critical bug fixes since version 3.1.2.