Logstash Lumberjack Cert

kharvey · April 1, 2019, 10:29pm

I have a pipeline that is designed like:
Filebeat -> Logstash A -> Logstash B -> Elastic Search

It took me a while but I figured out how to get communication up and running between Logstash A and Logstash B. But now I am having a certificate problem that I haven't been able to resolve.

Here are all of my respective setups:
Filebeat output:

output.logstash:
  # The Logstash hosts
  hosts: ["logstash-proxy.appliance.me.com:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/ssl/ca.me.com.crt"]
  ssl.certificate: "/etc/filebeat/ssl/logstash-client-pod.crt"
  ssl.key: "/etc/filebeat/ssl/logstash-client-pod.key"

Logstash A input:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"]
    ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
    ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
    ssl_verify_mode => "force_peer"
    add_field => {
      "_forwarder" => "fwd-5044"
      "origin_host" => "%{host}"
    }
  }
}

Logstash A output:

output {
  if [_forwarder] == "fwd-5044" {
    lumberjack {
      id => "proxy-5044"
      port => 5044
      hosts => "logging.client-appliance.me.com"
      codec => "json"
      ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
    }
  }
}

Logstash B input:

input {
  beats {
    port => 5044
    codec => json
    ssl => true
    ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
    ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
    ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"]
    ssl_verify_mode => "peer"
  }
}

When Logstash A connects to Logstash B I receive the following error:

[2019-04-01T15:02:49,032][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.1.2.101:38262] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
[2019-04-01T15:02:49,035][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN

I know it has to be a problem with the certs, as I can generate a lumberjack cert from the directions on the website everything works fine.

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.cert -subj /CN=localhost

The thing that is killing me is I can't figure out what is wrong with my certs. They are certs that I generated and chained together. They are the exact same on my two Logstash servers, but I keep getting the same error.

Any suggestions?

kharvey · April 2, 2019, 6:37pm

I have made it a little farther with my tests, but I still don't have a working solution. The problem appears to be with using csr and then signing that csr.

Here are the working commands to generate certs and keys that work with lumberjack:

openssl req -x509 -nodes -newkey rsa:2048 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf
openssl req -x509 -new -nodes -newkey rsa:4096 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf
openssl req -x509 -new -batch -nodes -newkey rsa:4096 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf
openssl req -x509 -new -batch -nodes -newkey rsa:4096 -keyout stupidtest.key -out stupidtest.crt -config CAConfig.conf
openssl genrsa -out stupidtest.key 4096
  openssl req -x509 -new -batch -nodes -key stupidtest.key -out stupidtest.crt -config CAConfig.conf
  openssl pkcs8 -in stupidtest.key -topk8 -out stupidtest-pkcs8.key -nocrypt

If I generate the key outside of the cert request you have to convert it to pkcs8 otherwise you get errors on Logstash for invalid certificate.
So all of the above commands will create working certs to use with Logstash and Lumberjack. Yes, I know I have some overkill with what I tested, but I didn't want to run into something dumb, so I iterated over everything.

Now on to the commands that do not work:

openssl genrsa -out stupidtest.key 4096
openssl req -new -batch -nodes -key stupidtest.key -out stupidtest.csr -config CAConfig.conf
openssl x509 -req -in stupidtest.csr -CA CA/CACert.pem -CAkey CA/CAKey.pem -CAcreateserial -out stupidtest.crt -days 2

Just in case there was a problem with my certificate authority cert, I generated a new one by running:

openssl req -x509 -new -nodes -key CAKey.pem -out me2.crt -config CAConfig.conf

I'm not sure where else to go from here. I may opt to create a bug in Github for this, unless anyone has any other suggestions.

kharvey · April 2, 2019, 7:22pm

Alright, I am pretty sure this is a bug in Lumberjack now.

I can run this without error:

openssl s_client -CAfile me2.crt -servername  logging.client-appliance.me.com -connect logging.client-appliance.me.com:5044-cert stupidtest.crt -key stupidtest.key

So I have working SSL using the certs that I have. The problem is Lumberjack is doing something that is breaking that communication. My assumption is it's because Lumberjack doesn't have the option for ssl_key, ssl_cacert with the ssl_certificate.

gruselglatz · April 3, 2019, 1:20pm

I am in the same situation as you. I configured everything like here: https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html

But i get sometimes PEER_DID_NOT_RETURN_A_CERTIFICATE or SSLV3_ALERT_CERTIFICATE_UNKNOWN

my configs:
upstream:

output {
  lumberjack {
    hosts => "agg_hag1.domain.com"
    port => 5055
    ssl_certificate => "/home/logstash/certs/agg_hag1.crt"
    codec => "json"
  }
}

downstream:

input {
  beats {
    port => 5055
    codec => "json"
    ssl => true
    ssl_certificate => "/home/logstash/certs/agg_hag1.crt"
    ssl_certificate_authorities => "/home/logstash/certs/domain-ca-pem.crt"
    ssl_key => "/home/logstash/certs/agg_hag1-des-v1.pem"
    ssl_key_passphrase => "PWD"
#    ssl_verify_mode => none
#    tls_min_version => 1.2
  }
}

guyboertje · April 3, 2019, 2:15pm

@kharvey @gruselglatz

I have found this blog post very very useful in the past.

https://benjaminknofe.com/blog/2018/07/08/logstash-authentication-with-ssl-certificates/

kharvey · April 3, 2019, 4:14pm

@guyboertje
Thanks for the link. That has some great tips. The problem is that Lumberjack doesn't use the same SSL configuration as beats does. All of my filebeat and winlogbeat have a working SSL configuration. I just can't get it to work with Lumberjack.

@gruselglatz
Try setting your ssl_verify_mode to "peer"

At this point I have dropped Lumberjack and I am going back to TCP with JSON_LINES. I am having to refactor my pipelines for the change to the output, but that will be way quicker than me troubleshooting farther. From what I can tell Lumberjack is a dying plugin.

gruselglatz · April 4, 2019, 5:05am

@kharvey I can feel your pain! I've read all Blogs, Issues on Github, even Ruby Source (God this is hard)....
The docs about Logstash and SSL are awful, i spent now 2 Weeks to get it up and nearly running
IDK where to go now, maybe I should switch to some other inter Logstash transport method. Did you get TCP with JSON_Lines and SSL up and running?

Would you be so kind and post a working config for this here, I am tired now

Thanks for you time and help!

OH and peer mode didnt make a difference

guyboertje · April 4, 2019, 9:40am

We know it is hard - the certificate creation steps are out of our hands. Elasticsearch provides a certutil tool - you may find this useful to automate the cert creation.

I have also read that it is important to have DNS lookups (/etc/hosts or server) that link IP and server name used for the -subj /CN=logstash.local
Alternatively you can use the IP but you need to set the subjectAltName = IP:192.168.1.10 in the [ v3_ca ] section of the openssl.cnf file (or equivalent).
See this guide

Lastly and for what it is worth, we have a BETA feature pipeline to pipeline communication, this is comms between two pipelines in the same Logstash instance. This may be useful and YMMV.

kharvey · April 4, 2019, 3:04pm

I don't believe the problem is the certs themselves. The problem is how Lumberjack handles the certs. Lumberjack does not appear to support chained certificates. I can get it running with self signed certs all day long. But if I try to do a CSR, it fails. I can use those same chained certs with the openssl client connections and everything works, but the second I switch over to Lumberjack they fail. So I know my certs work, especially since i use the same chained certs for all of my Filebeat and ElasticSearch communications.
I have tried DNS and IP address all with the same results.

Reading through the code for Lumberjack, it just doesn't have the SSL options that say TCP or Filebeats has.

@gruselglatz
Yes, I have all of my pipelines up and running now. I am testing today in dev for a possible deployment tonight. Here are my configs:

Filebeat Out:

 output.logstash:
  hosts: ["logstash-proxy.appliance.me.com:5044"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/filebeat/ssl/ca.me.com.crt"]
  ssl.certificate: "/etc/filebeat/ssl/logstash-client-pod.crt"
  ssl.key: "/etc/filebeat/ssl/logstash-client-pod.key"

Logstash A In:

input {
  beats {
    id => "proxy-5044-in"
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"]
    ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
    ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
    ssl_verify_mode => "force_peer"
    add_field => {
      "_forwarder" => "fwd-5044"
      "origin_host" => "%{host}"
    }
  }

Logstash A Out:

output {
  if [_forwarder] == "fwd-5044" {
    tcp {
      id => "proxy-5044-out"
      port => 5044
      host => "logging.client-appliance.me.com"
      codec => "json_lines"
      mode => "client"
      reconnect_interval => 1
      ssl_verify => true
      ssl_enable => true
      ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
      ssl_cert => "/etc/logstash/ssl/logstash-proxy.crt"
      ssl_cacert => "/etc/logstash/ssl/ca.me.com.crt"
    }
  }
}

Logstash B In:

input {
  tcp {
    port => 5044
    codec => "json_lines"
    ssl_enable => true
    ssl_verify => true
    ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
    ssl_cert => "/etc/logstash/ssl/logstash-proxy.crt"
    ssl_extra_chain_certs => "/etc/logstash/ssl/ca.me.com.crt"    
  }
}

This is a fully functional pipeline with SSL. The only downside to doing it with TCP is that there is no compression. I don't know if that will be a problem for my setup yet or not. Like I said I still need to test in dev.

If you need help creating your certs as well, the document that @guyboertje posted above is a great example. I can also make some suggestions as well if you need help with that.
Also for what it is worth, the pipeline to pipeline communication is awesome. I don't think you would be able to use it in this scenario, but I use it to route all of my pipelines on Logstash.

kharvey · April 26, 2019, 9:58pm

@guyboertje and @gruselglatz

I found a place where it says that Lumberjack is deprecated.

I finally hit a point where I have to use Lumberjack so I am going to attempt to try and find and or fix Lumberjack cert problems so that it works.

system · May 24, 2019, 9:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.