I have a pipeline that is designed like:
Filebeat -> Logstash A -> Logstash B -> Elastic Search
It took me a while but I figured out how to get communication up and running between Logstash A and Logstash B. But now I am having a certificate problem that I haven't been able to resolve.
Here are all of my respective setups:
Filebeat output:
output.logstash:
# The Logstash hosts
hosts: ["logstash-proxy.appliance.me.com:5044"]
ssl.certificate_authorities: ["/etc/filebeat/ssl/ca.me.com.crt"]
ssl.certificate: "/etc/filebeat/ssl/logstash-client-pod.crt"
ssl.key: "/etc/filebeat/ssl/logstash-client-pod.key"
Logstash A input:
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"]
ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
ssl_verify_mode => "force_peer"
add_field => {
"_forwarder" => "fwd-5044"
"origin_host" => "%{host}"
}
}
}
Logstash A output:
output {
if [_forwarder] == "fwd-5044" {
lumberjack {
id => "proxy-5044"
port => 5044
hosts => "logging.client-appliance.me.com"
codec => "json"
ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
}
}
}
Logstash B input:
input {
beats {
port => 5044
codec => json
ssl => true
ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"]
ssl_verify_mode => "peer"
}
}
When Logstash A connects to Logstash B I receive the following error:
[2019-04-01T15:02:49,032][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.1.2.101:38262] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
[2019-04-01T15:02:49,035][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
I know it has to be a problem with the certs, as I can generate a lumberjack cert from the directions on the website everything works fine.
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.cert -subj /CN=localhost
The thing that is killing me is I can't figure out what is wrong with my certs. They are certs that I generated and chained together. They are the exact same on my two Logstash servers, but I keep getting the same error.
Any suggestions?