I'm stuck on the following issue and could use some help.
I'm attempting to send messages from one logstash instance to another via lumberjack. This worked perfectly in the past, but after updating logstash and its plugins, I'm no longer able to get it working. I get the following log when I start up the sending logstash instance:
All hosts unavailable, sleeping", :hosts=>["1.2.3.4"], :e=>#<OpenSSL::SSL::SSLError: certificate verify failed>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:190:in connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:87:inconnection_start'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:70:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:33:inconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.24/lib/lumberjack/client.rb:22:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-lumberjack-1.0.2/lib/logstash/outputs/lumberjack.rb:92:inconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-lumberjack-1.0.2/lib/logstash/outputs/lumberjack.rb:51:in register'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:164:in start_outputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:83:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/agent.rb:155:in execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:90:inrun'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:95:inrun'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/task.rb:12:ininitialize'"], :level=>:error}
What confuses me is the fact that logstash-forwarder (version 0.4.0) is able to validate the certificate and connects to the logstash receiving instance with no problems.
Also, since the above error says it's having problems verifying the certificate, I checked with openssl:
[root@myhost conf.d]# openssl verify -CAfile logstash.ca myhost.cert
myhost.cert: OK
Relevant Background Info:
- both hosts are RHEL 6.7 and fully patched / up to date
- Logstash 1.5.4
- logstash-output-plugin 1.0.2
- Java "1.8.0_51"
- Certificates are signed by a commercial CA.
- ssl_certificate (in logstash output) is set to logstash.ca. logstash.ca contains 2 intermediate certs and the root signing cert in PEM format, in reverse order. Exactly like one wold use when configuring apache for TLS.
Sending logstash instance config snippet:
lumberjack {
hosts => ['receiving']
port => 5000
ssl_certificate => "/etc/pki/tls/certs/logstash.ca"
codec => json { }
}
Receiving logstash instance config snippet:
lumberjack {
port => 5000
ssl_certificate => "/etc/pki/tls/certs/myhost.cert"
ssl_key => "/etc/pki/tls/private/myhost.key"
tags => ["lumberjack"]
}
}
I'm totally stuck here, so any help or idea would be greatly appreciated.