Logstash 1.5.4 - lumberjack-output SSL verification error

Elastic Stack Logstash
1

I'm stuck on the following issue and could use some help.

I'm attempting to send messages from one logstash instance to another via lumberjack. This worked perfectly in the past, but after updating logstash and its plugins, I'm no longer able to get it working. I get the following log when I start up the sending logstash instance:

All hosts unavailable, sleeping", :hosts=>["1.2.3.4"], :e=>#<OpenSSL::SSL::SSLError: certificate verify failed>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:190:in connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:87:inconnection_start'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:70:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:33:inconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:22:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-lumberjack-1.0.2/lib/logstash/outputs/lumberjack.rb:92:inconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-lumberjack-1.0.2/lib/logstash/outputs/lumberjack.rb:51:in register'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:164:in start_outputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:83:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/agent.rb:155:in execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:90:inrun'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:95:inrun'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/task.rb:12:ininitialize'"], :level=>:error}

What confuses me is the fact that logstash-forwarder (version 0.4.0) is able to validate the certificate and connects to the logstash receiving instance with no problems.

Also, since the above error says it's having problems verifying the certificate, I checked with openssl:

[root@myhost conf.d]# openssl verify -CAfile logstash.ca myhost.cert
myhost.cert: OK

Relevant Background Info:

  • both hosts are RHEL 6.7 and fully patched / up to date
  • Logstash 1.5.4
  • logstash-output-plugin 1.0.2
  • Java "1.8.0_51"
  • Certificates are signed by a commercial CA.
  • ssl_certificate (in logstash output) is set to logstash.ca. logstash.ca contains 2 intermediate certs and the root signing cert in PEM format, in reverse order. Exactly like one wold use when configuring apache for TLS.

Sending logstash instance config snippet:

lumberjack {
hosts => ['receiving']
port => 5000
ssl_certificate => "/etc/pki/tls/certs/logstash.ca"
codec => json { }
}

Receiving logstash instance config snippet:
lumberjack {
port => 5000
ssl_certificate => "/etc/pki/tls/certs/myhost.cert"
ssl_key => "/etc/pki/tls/private/myhost.key"
tags => ["lumberjack"]
}
}

I'm totally stuck here, so any help or idea would be greatly appreciated.

2

Try setting debug on the sending LS to see what is happening? Also check if you can telnet to the receiving host on port 5000 from the sending host.