# Logstash Lumberjack Cert **URL:** \ **Category:** Logstash\ **Created:** [April 1, 2019, 10:29pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885 "2019-04-01T22:29:21Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 1, 2019, 10:29pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/1 "2019-04-01T22:29:21Z")
I have a pipeline that is designed like: Filebeat -\> Logstash A -\> Logstash B -\> Elastic Search It took me a while but I figured out how to get communication up and running between Logstash A and Logstash B. But now I am having a certificate problem that I haven't been able to resolve. Here are all of my respective setups: Filebeat output: ```auto output.logstash: # The Logstash hosts hosts: ["logstash-proxy.appliance.me.com:5044"] ssl.certificate_authorities: ["/etc/filebeat/ssl/ca.me.com.crt"] ssl.certificate: "/etc/filebeat/ssl/logstash-client-pod.crt" ssl.key: "/etc/filebeat/ssl/logstash-client-pod.key" ``` Logstash A input: ```auto input { beats { port => 5044 ssl => true ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"] ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt" ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_verify_mode => "force_peer" add_field => { "_forwarder" => "fwd-5044" "origin_host" => "%{host}" } } } ``` Logstash A output: ```auto output { if [_forwarder] == "fwd-5044" { lumberjack { id => "proxy-5044" port => 5044 hosts => "logging.client-appliance.me.com" codec => "json" ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt" } } } ``` Logstash B input: ```auto input { beats { port => 5044 codec => json ssl => true ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt" ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"] ssl_verify_mode => "peer" } } ``` When Logstash A connects to Logstash B I receive the following error: ```auto [2019-04-01T15:02:49,032][INFO][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.1.2.101:38262] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN [2019-04-01T15:02:49,035][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) ~[netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN ``` I know it has to be a problem with the certs, as I can generate a lumberjack cert from the directions on the website everything works fine. ```auto openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.cert -subj /CN=localhost ``` The thing that is killing me is I can't figure out what is wrong with my certs. They are certs that I generated and chained together. They are the exact same on my two Logstash servers, but I keep getting the same error. Any suggestions? ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 2, 2019, 6:37pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/2 "2019-04-02T18:37:31Z")
I have made it a little farther with my tests, but I still don't have a working solution. The problem appears to be with using csr and then signing that csr. Here are the working commands to generate certs and keys that work with lumberjack: ```auto openssl req -x509 -nodes -newkey rsa:2048 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf openssl req -x509 -new -nodes -newkey rsa:4096 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf openssl req -x509 -new -batch -nodes -newkey rsa:4096 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf openssl req -x509 -new -batch -nodes -newkey rsa:4096 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf openssl genrsa -out stupidtest.key 4096 openssl req -x509 -new -batch -nodes -key stupidtest.key -out stupidtest.crt -config CAConfig.conf openssl pkcs8 -in stupidtest.key -topk8 -out stupidtest-pkcs8.key -nocrypt ``` If I generate the key outside of the cert request you have to convert it to pkcs8 otherwise you get errors on Logstash for invalid certificate. So all of the above commands will create working certs to use with Logstash and Lumberjack. Yes, I know I have some overkill with what I tested, but I didn't want to run into something dumb, so I iterated over everything. Now on to the commands that do not work: ```auto openssl genrsa -out stupidtest.key 4096 openssl req -new -batch -nodes -key stupidtest.key -out stupidtest.csr -config CAConfig.conf openssl x509 -req -in stupidtest.csr -CA CA/CACert.pem -CAkey CA/CAKey.pem -CAcreateserial -out stupidtest.crt -days 2 ``` Just in case there was a problem with my certificate authority cert, I generated a new one by running: ```auto openssl req -x509 -new -nodes -key CAKey.pem -out me2.crt -config CAConfig.conf ``` I'm not sure where else to go from here. I may opt to create a bug in Github for this, unless anyone has any other suggestions. ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 2, 2019, 7:22pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/3 "2019-04-02T19:22:19Z")
Alright, I am pretty sure this is a bug in Lumberjack now. I can run this without error: ```auto openssl s_client -CAfile me2.crt -servername logging.client-appliance.me.com -connect logging.client-appliance.me.com:5044-cert stupidtest.crt -key stupidtest.key ``` So I have working SSL using the certs that I have. The problem is Lumberjack is doing something that is breaking that communication. My assumption is it's because Lumberjack doesn't have the option for ssl\_key, ssl\_cacert with the ssl\_certificate. ---
**Author:** ![gruselglatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gruselglatz/32/42675_2.png) [@gruselglatz](https://discuss.elastic.co/u/gruselglatz)\ **Post date:** [April 3, 2019, 1:20pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/4 "2019-04-03T13:20:42Z")
I am in the same situation as you. I configured everything like here: [https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html](https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html) But i get sometimes PEER\_DID\_NOT\_RETURN\_A\_CERTIFICATE or SSLV3\_ALERT\_CERTIFICATE\_UNKNOWN my configs: upstream: ``` output { lumberjack { hosts => "agg_hag1.domain.com" port => 5055 ssl_certificate => "/home/logstash/certs/agg_hag1.crt" codec => "json" } } ``` downstream: ``` input { beats { port => 5055 codec => "json" ssl => true ssl_certificate => "/home/logstash/certs/agg_hag1.crt" ssl_certificate_authorities => "/home/logstash/certs/domain-ca-pem.crt" ssl_key => "/home/logstash/certs/agg_hag1-des-v1.pem" ssl_key_passphrase => "PWD" # ssl_verify_mode => none # tls_min_version => 1.2 } } ``` ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 3, 2019, 2:15pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/5 "2019-04-03T14:15:06Z")
@kharvey @gruselglatz I have found this blog post very very useful in the past. [https://benjaminknofe.com/blog/2018/07/08/logstash-authentication-with-ssl-certificates/](https://benjaminknofe.com/blog/2018/07/08/logstash-authentication-with-ssl-certificates/) ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 3, 2019, 4:14pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/6 "2019-04-03T16:14:08Z")
@guyboertje Thanks for the link. That has some great tips. The problem is that Lumberjack doesn't use the same SSL configuration as beats does. All of my filebeat and winlogbeat have a working SSL configuration. I just can't get it to work with Lumberjack. @gruselglatz Try setting your ssl\_verify\_mode to "peer" At this point I have dropped Lumberjack and I am going back to TCP with JSON\_LINES. I am having to refactor my pipelines for the change to the output, but that will be way quicker than me troubleshooting farther. From what I can tell Lumberjack is a dying plugin. ---
**Author:** ![gruselglatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gruselglatz/32/42675_2.png) [@gruselglatz](https://discuss.elastic.co/u/gruselglatz)\ **Post date:** [April 4, 2019, 5:05am UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/7 "2019-04-04T05:05:45Z")
@kharvey I can feel your pain! I've read all Blogs, Issues on Github, even Ruby Source (God this is hard).... The docs about Logstash and SSL are awful, i spent now 2 Weeks to get it up and nearly running 😫 IDK where to go now, maybe I should switch to some other inter Logstash transport method. Did you get TCP with JSON\_Lines and SSL up and running? Would you be so kind and post a working config for this here, I am tired now ☹ Thanks for you time and help! OH and peer mode didnt make a difference 😐 ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 4, 2019, 9:40am UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/8 "2019-04-04T09:40:20Z")
We know it is hard - the certificate creation steps are out of our hands. Elasticsearch provides a [certutil](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/certutil.html) tool - you may find this useful to automate the cert creation. I have also read that it is important to have DNS lookups (/etc/hosts or server) that link IP and server name used for the `-subj /CN=logstash.local` Alternatively you can use the IP but you need to set the `subjectAltName = IP:192.168.1.10` in the `[v3_ca]` section of the openssl.cnf file (or equivalent). [See this guide](https://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-18-04-ubuntu-16-04.html#Create_SSL_certificate_for_Logstash_Optional) Lastly and for what it is worth, we have a BETA feature [pipeline to pipeline communication](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html), this is comms between two pipelines in the same Logstash instance. This may be useful and YMMV. ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 4, 2019, 3:04pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/9 "2019-04-04T15:04:17Z")
I don't believe the problem is the certs themselves. The problem is how Lumberjack handles the certs. Lumberjack does not appear to support chained certificates. I can get it running with self signed certs all day long. But if I try to do a CSR, it fails. I can use those same chained certs with the openssl client connections and everything works, but the second I switch over to Lumberjack they fail. So I know my certs work, especially since i use the same chained certs for all of my Filebeat and ElasticSearch communications. I have tried DNS and IP address all with the same results. Reading through the code for Lumberjack, it just doesn't have the SSL options that say TCP or Filebeats has. @gruselglatz Yes, I have all of my pipelines up and running now. I am testing today in dev for a possible deployment tonight. Here are my configs: Filebeat Out: ```auto output.logstash: hosts: ["logstash-proxy.appliance.me.com:5044"] ssl.enabled: true ssl.certificate_authorities: ["/etc/filebeat/ssl/ca.me.com.crt"] ssl.certificate: "/etc/filebeat/ssl/logstash-client-pod.crt" ssl.key: "/etc/filebeat/ssl/logstash-client-pod.key" ``` Logstash A In: ```auto input { beats { id => "proxy-5044-in" port => 5044 ssl => true ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"] ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt" ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_verify_mode => "force_peer" add_field => { "_forwarder" => "fwd-5044" "origin_host" => "%{host}" } } ``` Logstash A Out: ```auto output { if [_forwarder] == "fwd-5044" { tcp { id => "proxy-5044-out" port => 5044 host => "logging.client-appliance.me.com" codec => "json_lines" mode => "client" reconnect_interval => 1 ssl_verify => true ssl_enable => true ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_cert => "/etc/logstash/ssl/logstash-proxy.crt" ssl_cacert => "/etc/logstash/ssl/ca.me.com.crt" } } } ``` Logstash B In: ```auto input { tcp { port => 5044 codec => "json_lines" ssl_enable => true ssl_verify => true ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_cert => "/etc/logstash/ssl/logstash-proxy.crt" ssl_extra_chain_certs => "/etc/logstash/ssl/ca.me.com.crt" } } ``` This is a fully functional pipeline with SSL. The only downside to doing it with TCP is that there is no compression. I don't know if that will be a problem for my setup yet or not. Like I said I still need to test in dev. If you need help creating your certs as well, the document that @guyboertje posted above is a great example. I can also make some suggestions as well if you need help with that. Also for what it is worth, the pipeline to pipeline communication is awesome. I don't think you would be able to use it in this scenario, but I use it to route all of my pipelines on Logstash. ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 26, 2019, 9:58pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/10 "2019-04-26T21:58:36Z")
@guyboertje and @gruselglatz I found a place where it says that Lumberjack is deprecated. > **[elastic/ruby-lumberjack](https://github.com/elastic/ruby-lumberjack)** > > Ruby lumberjack protocol, contains both client and server - elastic/ruby-lumberjack I finally hit a point where I have to use Lumberjack so I am going to attempt to try and find and or fix Lumberjack cert problems so that it works. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 24, 2019, 9:58pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-cert/174885/11 "2019-05-24T21:58:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.