We are implementing application log monitoring on ELK flow from filebeat (logstash output) -> on-prem logstash server(beats input & lumberjack output) -> aws logstash server (beats input & elastic output) -> elasticearch. The communication between on-prem logstash to aws logstash is ssl enabled and we are getting below error for certificate verification. We are using signed certificate with domain based on both lostash.
[2021-01-11T01:56:51,653][ERROR][logstash.outputs.lumberjack][onprem-nonpcf-filebeat] All hosts unavailable, sleeping {:hosts=>["x.x.x.x"], :e=>#<OpenSSL::SSL::SSLError: certificate verify failed>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:266:in connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:95:in connection_start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:76:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:34:in connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:24:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:86:in connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:49:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:126:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:216:in block in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:215:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:520:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125:in block in start'"]}
On-prem Logstash output Configuration:
lumberjack {
id => "onprem-l1"
codec => "json"
port => 5045
hosts => ["awslogstash.prod.domain.com"] #domain name renamed
ssl_certificate => "/etc/logstash/certs/ca. crt" # which contains root ca and intermedia certificate
}
We tried configuring domain certificate, root ca certificate, intermediate certificate on ssl_certificate option with seprate and combined but getting same validation error.
AWS Logstash input configuration:
input {
beats {
host => " awslogstash.prod.domain.com"
port => 5045
ssl => true
ssl_certificate => "/etc/logstash/certs/ awslogstash.prod.domain.com.crt"
ssl_key => "/etc/logstash/certs/all-nodes.pkcs8.key"
ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
ssl_verify_mode => "peer"
}
}
Please let me know what I am missing.
Thanks,
Alex