Filebeat failed to send message to logstash with SSL enabled

Zou_Panxia · August 16, 2016, 11:29am

I downloaded the latest logstash(2.3.4) and filebeat (1.2.3) for some testing. It works when ssl is not enabled. When it is enabled with ssl, curl verification works as well. However filebeat keeps to report "ERR SSL client failed to connect with: EOF" and there is no any output on logstash side even with -v. Anybody can help? Thanks!

#for logstash

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/ssl/certs/logstash.pem"
    ssl_key => "/etc/ssl/private/logstash.key"
  }
}
output {
  stdout { codec => rubydebug }
}

#filebeat.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.log
output:
  logstash:
    hosts: ["localserver:5044"]
    tls:
      certificate_authorities: [ "/usr/local/share/ca-certificates/logging-forward.crt" ]

#works with curl (just some snippet of result)

curl -v --cacert  /usr/local/share/ca-certificates/logging-forward.crt.bk https://localserver:5044
* Rebuilt URL to: https://localserver:5044/
* Hostname was NOT found in DNS cache
*   Trying 169.44.130.118...
...
*        SSL certificate verify ok.
...
* SSLv3, TLS alert, Client hello (1):
* Empty reply from server
* Connection #0 to host localserver left intact
curl: (52) Empty reply from server

#transport.go:125: ERR SSL client failed to connect with: EOF

[root@ucd filebeat]# filebeat -c filebeat.yml -e -v      
...
2016/08/16 11:17:07.167210 transport.go:125: ERR SSL client failed to connect with: EOF
2016/08/16 11:17:07.167233 outputs.go:126: INFO Activated logstash as output plugin.
2016/08/16 11:17:07.167413 publish.go:288: INFO Publisher name: ucd.novalocal
.....
2016/08/16 11:17:07.171399 crawler.go:78: INFO All prospectors initialised with 6 states to persist
2016/08/16 11:17:07.171414 registrar.go:87: INFO Starting Registrar
2016/08/16 11:17:07.171430 publish.go:88: INFO Start sending events to output
2016/08/16 11:17:09.684649 transport.go:125: ERR SSL client failed to connect with: EOF
2016/08/16 11:17:09.684669 single.go:126: INFO Connecting error publishing events (retrying): EOF
2016/08/16 11:17:09.684688 single.go:152: INFO send fail
2016/08/16 11:17:09.684696 single.go:159: INFO backoff retry: 1s
...

#tcpdump on logstash (logstash is on a VM with floating ip):

10:50:35.531981 IP 169.44.130.10.41782 > 192.168.1.152.lxi-evntsvc: Flags [S], seq 1955704033, win 29200, options [mss 1460,sackOK,TS val 321287987 ecr 0,nop,wscale 7], length 0
10:50:35.532057 IP 192.168.1.152.lxi-evntsvc > 169.44.130.10.41782: Flags [S.], seq 3542285397, ack 1955704034, win 28960, options [mss 1460,sackOK,TS val 2721293661 ecr 321287987,nop,wscale 7], length 0
10:50:35.532565 IP 169.44.130.10.41782 > 192.168.1.152.lxi-evntsvc: Flags [.], ack 1, win 229, options [nop,nop,TS val 321287987 ecr 2721293661], length 0
10:50:35.532723 IP 169.44.130.10.41782 > 192.168.1.152.lxi-evntsvc: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 321287987 ecr 2721293661], length 155
10:50:35.532739 IP 192.168.1.152.lxi-evntsvc > 169.44.130.10.41782: Flags [.], ack 156, win 235, options [nop,nop,TS val 2721293661 ecr 321287987], length 0
10:50:35.537269 IP 192.168.1.152.lxi-evntsvc > 169.44.130.10.41782: Flags [F.], seq 1, ack 156, win 235, options [nop,nop,TS val 2721293666 ecr 321287987], length 0
10:50:35.537754 IP 169.44.130.10.41782 > 192.168.1.152.lxi-evntsvc: Flags [F.], seq 156, ack 2, win 229, options [nop,nop,TS val 321287988 ecr 2721293666], length 0
10:50:35.537776 IP 192.168.1.152.lxi-evntsvc > 169.44.130.10.41782: Flags [.], ack 157, win 235, options [nop,nop,TS val 2721293666 ecr 321287988], length 0

#tcpdump on filebeat (filebeat is on a VM with floating ip)

10:50:35.530927 IP 192.168.1.25.41782 > 169.44.130.118.5044: Flags [S], seq 1955704033, win 29200, options [mss 1460,sackOK,TS val 321287987 ecr 0,nop,wscale 7], length 0
10:50:35.531606 IP 169.44.130.118.5044 > 192.168.1.25.41782: Flags [S.], seq 3542285397, ack 1955704034, win 28960, options [mss 1460,sackOK,TS val 2721293661 ecr 321287987,nop,wscale 7], length 0
10:50:35.531681 IP 192.168.1.25.41782 > 169.44.130.118.5044: Flags [.], ack 1, win 229, options [nop,nop,TS val 321287987 ecr 2721293661], length 0
10:50:35.531848 IP 192.168.1.25.41782 > 169.44.130.118.5044: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 321287987 ecr 2721293661], length 155
10:50:35.532132 IP 169.44.130.118.5044 > 192.168.1.25.41782: Flags [.], ack 156, win 235, options [nop,nop,TS val 2721293661 ecr 321287987], length 0
10:50:35.536753 IP 169.44.130.118.5044 > 192.168.1.25.41782: Flags [F.], seq 1, ack 156, win 235, options [nop,nop,TS val 2721293666 ecr 321287987], length 0
10:50:35.536832 IP 192.168.1.25.41782 > 169.44.130.118.5044: Flags [F.], seq 156, ack 2, win 229, options [nop,nop,TS val 321287988 ecr 2721293666], length 0
10:50:35.537209 IP 169.44.130.118.5044 > 192.168.1.25.41782: Flags [.], ack 157, win 235, options [nop,nop,TS val 2721293666 ecr 321287988], length 0

steffens · August 16, 2016, 8:29pm

Can you run the curl test with tls version 1.1 (use --tlsv1.1)? By default beats require minimum tls 1.1 by default (I think), but version is configurable.

Please use </> button to configure logs/sources/config files.

steffens · August 16, 2016, 8:33pm

How did you create the certificates? Is logging-forward.crt same file as logstash.pem?

Zou_Panxia · August 17, 2016, 3:06am

Thanks Steffen.

I found that curl test works with --tlsv1 and --tlsv1.0, but it doesn't work with --tlsv1.1, --tlsv1.2. If Beat only supports tlsv1.1 as minimum version, that would be an issue. What is logs/sources/config exactly to change the requirement for filebeat to allow tlsv1.0? Thanks.

I also tried to update filebeat.yml to specify min_version and max_version, but looks there is no any effect.

steffens · August 17, 2016, 10:15am

set min_version to 1.0 or SSL-3.0.

Can you share modified config file + log output with debug enabled?

How did you create the certificates? Is logging-forward.crt same file as logstash.pem?

Zou_Panxia · August 17, 2016, 5:08pm

I set min_version to SSL-3.0 and didn't get it resolved. logging-forward.crt is different from logstash.pem. It is a chain cert.

I have installed logstash on CentOS 7.2. The wired thing today is that I tried to install it in Ubuntu, and it works with the complete cert/key/chain cert and the same logstash version on Trusy. The only difference could be openssl/jruby/java. After comparison, it is identified that jre is the cause. After installing openjre on CentOS, it started to work to receive message from filebeat.

Thanks your help.

#old configuration with SSL problem
java -jar vendor/jruby/lib/jruby.jar -v
jruby 1.7.23 (1.9.3p551) 2015-11-24 f496dd5 on IBM J9 VM pxa6480sr3fp10-20160720_02 (SR3 FP10) +jit [linux-amd64]

#new configuration with openssl
java -jar vendor/jruby/lib/jruby.jar -v
jruby 1.7.23 (1.9.3p551) 2015-11-24 f496dd5 on OpenJDK 64-Bit Server VM 1.8.0_101-b13 +jit [linux-amd64]

system · September 6, 2016, 11:29am

This topic was automatically closed after 21 days. New replies are no longer allowed.