Logstash-input-beats - File does not contain valid private key

Elastic Stack Logstash
1

Hi there,

I am currently doing a stack upgrade from 7.0.1 to 7.6.2. Everything went fine on the Elasticsearch cluster and Kibana instances but I have issues upgrading Logstash.

To be precise, Logstash upgrade was fine, the issue I have is located in a pipeline and precisely in a Beats input (so logstash-input-beats).

The problem seems to be located in the ssl_key param of the beats input. The following error is popping up in Logtash logs:

[2020-04-15T14:07:44,572][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"mon_collector", :exception=>java.lang.IllegalArgumentException: File does not contain valid private key: /etc/pki/local/my-host.key.pem, :backtrace=>["io.netty.handler.ssl.SslContextBuilder.keyManager(io/netty/handler/ssl/SslContextBuilder.java:270)", "io.netty.handler.ssl.SslContextBuilder.forServer(io/netty/handler/ssl/SslContextBuilder.java:90)", "org.logstash.netty.SslContextBuilder.buildContext(org/logstash/netty/SslContextBuilder.java:104)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)"

What is strange is that everything was working fine with Logstash 7.0.1 and I did not change anything to the configuration / pipeline configuration during the upgrade. Moreover, I tried to rollback on 7.0.1 it worked fine again, I also tried version 7.4.2, 7.5.2, it is working fine too...

The breaking point seems to be version 7.6.x (I tried 7.6.1, 7.6.2, both output the error and the pipeline does not start). I did not try 7.6.0 but I think it will be the same...

I searched a lot (here, GitHub, StackOverflow, googling) and tried about everything I found:

  • File permission (including changing owner to Logstash)
  • Ensure private key format is correct (by the way a copy of the same private key is used by Elasticsearch on this host and is currently working with 7.6.2)
  • Generate a new key

At this point, I believe this is a specific issue regarding 7.6.x version(s), and since I did not find any recent post this might be the case (7.6.x are recent versions at the time I'm writing this).

Again, what is really strange is that 7.6 upgrade seems to break things. I have seen in release notes that there were some changes around SSL things (like netty) recently and may be it is related.

So here is my main question: while upgrading from 7.0.1 (or 7.5.2) to 7.6.2, am I supposed to do anything particular on the SSL settings?

Thank you in advance for your help,
Kuaaaly

1 Like
2

I would appreciate if you can provide any insights about this @RobBavey?
Much appreciated!

3

Digging into this a bit further...

Logstash 7.5.2 uses version 6.0.5 of the logstash-input-beats plugin
Logstash 7.6.0 uses version 6.0.6 of the logstash-input-beats plugin

Looking at the commits of version 6.0.5 (used by Logstash 7.5.2), we can see that the plugin used the following version of the Netty library:

io.netty:netty-all:4.1.44.Final

Looking at the commits for version 6.0.6 (just after the 7.5.2 stack release), we can see that the Netty library version changed to:

io.netty:netty-all:4.1.34.Final

That change was due to another regression, but it seemed to also have an impact on the SSL settings that might cause the above issue.

Also worth noting that before 6.0.5, the logstash-input-beats plugin was using the following version without any issue:

io.netty:netty-all:4.1.30.Final

So given that

  • 4.1.30 worked (pre-7.5.2)
  • 4.1.44 worked (7.5.2)
  • 4.1.34 doesn't work (post-7.5.2)

There must have been a change in Netty between 4.1.34 and 4.1.44 that causes this issue.

4

@kuaaly Can you give a little more information?

How did you generate the key? Is it protected with a password? Can you share your configuration, redacting any sensitive information that may be in there?

Regards,

Rob

5

Thanks for chiming in Rob!

The key has been generated using this command and is not protected by any password

openssl genrsa -out host1.key.pem 2048

The configuration is pretty simple:

beats {
    port => 1234
    ssl => true
    ssl_certificate => "/path/to/crt.pem"
    ssl_certificate_authorities => "/path/to/ca"
    ssl_key => "/path/to/host1.key.pem"
    tls_min_version => 1.1
    client_inactivity_timeout => 180
}
6

@val Thanks for the update -

The key needs to be in pkcs8 format, rather than pem - see the documentation for the beats input. Can you try that and see if you still see the error?

Thanks,

Rob

7

Hi Rob, thanks for your input.

As far as the openssl documentation goes, what comes out of that command should be a PEM-encoded PKCS#8 RSA key. Moreover, I insist on the fact that the same key was previously working on 7.0.1, 7.4.2, and 7.5.2 versions. I also looked at the documentation page for those versions of Logstash and they have the same pre-requisites that make me believe that our key might not be the problem.

Despite these findings, I tried the following:
Converting the key using this command:

openssl pkcs8 -topk8 -inform pem -in our.key.pem -outform pem -nocrypt -out new.key.pem

Also tried this:

openssl rsa -in our.key.pem -text > temp.key.pem
openssl pkcs8 -topk8 -inform pem -in temp.key.pem -outform pem -nocrypt -out new.key.pem

The second case bring the key back to traditional format (I have checked it manually with a cat between the two commands), before converting it in a PEM-encoded PKCS#8 RSA key.

None of those tries solved the problem. We still have exactly the same issue / error in the logs. Moreover, we tried last week to launch Logstash with the debug flag and this doesn't prompt anything more.

Last but not the least. As proposed by the support, we tried putting ssl_key_passphrase => '' in the input configuration, nothing changed.

Anyone here able to reproduce this issue on a fresh install? 7.6.x releases are pretty recent but I would be really surprised if we were the only ones facing this issue...

2 Likes
8

Hey, I tried reproducing but somehow all seems to be working (wout password protection).
For simplicity, I've generated a cert + key pair using :

openssl req -x509 -nodes -newkey rsa:4096 -keyout a_key.pem -out a_cert.pem -days 365

Than setup LS beats input and file-beats using the created ssl_certificate and ssl_key.
I am using LS 7.6.2 (the ssl_key_passphrase => '' isn't needed).

Probably missed something obvious as this was supposed to only work in LS 7.5 but not 7.6, right?

9

When testing hypothesis, please keep in mind that, before the beats input 6.0.8 we didn't put the key itself to use unless a client connected to logstash. This means that just starting logstash isn't enough of a test when comparing Logstash 7.6.1 and previous versions.

In this thread, @val I'm not sure from your analysis, but have you observed a situation where a private key works with logstash 7.5.2 but doesn't with 7.6.0? If so was this key PKCS1 or 8? and if it was 8, was it encrypted?

10

@jsvd thanks for chiming in.

What's important to note here is that it works on 7.0.1 but stops working on 7.6.0 without changing anything else than the stack version. We did not regenerate any key or certificate or anything.

If we start any version of Logstash (7.0.1 up to 7.5.2), events are flowing through the beats input without any issue as far as SSL is concerned.
As soon as we start Logstash 7.6 (7.6.0 up to 7.6.2), we start getting the issue mentioned in the initial post in this thread.

So our observation is that it seems quite obvious that something changed in the beats input between 7.5.2 and 7.6.0, and judging by the different Netty library upgrades/downgrades that occurred between those versions, I would not rule that out.

11

What's important to note here is that it works on 7.0.1 but stops working on 7.6.0 without changing anything else than the stack version . We did not regenerate any key or certificate or anything.

That is indeed something we're trying to solve and I agree it's a bug, but we've been struggling to have a reproducer. We can check that pkcs1 pkeys don't work in either version (even if logstash 7.0.1 boots up, but prints the same error once a beats connects).

Can you confirm that the key that worked before was pkcs1? Did you also use the same key in a tcp input in the same pipeline?

I'm hunting an hypothesis that the bouncycastleprovider is being added to java.security in some situations which makes pkcs1 work. your feedback here helps! thanks

12

I am far far away from a SSL expert and I think I found wrong informations by googling about openssl and unlike what I said in my previous post it seems that using openssl genrsa -out host1.key.pem 2048 command to generate the key gives a PKCS#1 and not PKCS#8. So our key is in PCKS#1 format and not PKCS#8 (header: BEGIN RSA PRIVATE KEY).

@jsvd, I also had this hypothesis during my first researches but I quickly discard it because of the beats input documentation (regarding private key format requirement). So I think your previous message brings back this hypothesis.

So what's (should be) sure now:

  • Logstash 7.5.2 is running smoothly with a PCKS#1 key (so 7.0.1 & 7.4.2 does). Events are passing.
  • Beats input plugin fail to start with Logstash 7.6.2 (with a PCKS#1 key)
  • We converted manually the PCKS#1 key to PKCS#8 using the command in this post (BEGIN PRIVATE KEY), Logstash is starting put the beats input fails to start again with the error copied in initial post.

About a tcp input plugin, sadly we won't be able to try it.

13

Thanks for the feedback @quentin.legraverend. I've been trying to replicate this but as long as I use a pkcs#1 key I'm not able to use it in Logstash beats input, neither 7.0.1 nor 7.6.2.

Here is the Dockerfile I'm testing with:

FROM logstash:7.0.1
USER root
RUN yum install -y openssl
USER logstash
RUN openssl genrsa -out key.pem 2048
RUN openssl req -x509 -new -key key.pem -out cert.pem -days 365 -subj '/CN=localhost' -nodes
RUN printf 'input {\n'                                            > pipeline/logstash.conf
RUN printf '  beats {\n'                                            >> pipeline/logstash.conf
RUN printf '    port => 1111\n'                                     >> pipeline/logstash.conf
RUN printf '    ssl => true\n'                                        >> pipeline/logstash.conf
RUN printf '    ssl_certificate => "/usr/share/logstash/cert.pem"\n'  >> pipeline/logstash.conf
RUN printf '    ssl_key => "/usr/share/logstash/key.pem"\n'          >> pipeline/logstash.conf
RUN printf '}}\n'                                                 >> pipeline/logstash.conf
RUN printf 'output { stdout {} }'                               >> pipeline/logstash.conf

You'll notice that docker build . -t mytest:7.0.1 && docker run -p 1111:1111 -it mytest:7.0.1 will start correctly, but if you connect a beat to 127.0.0.1:1111 logstash will refuse the connection with an error:

[2020-04-22T15:48:40,386][INFO ][org.logstash.beats.Server] Starting server on port: 1111
[2020-04-22T15:48:40,797][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-04-22T15:49:03,887][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/logstash/key.pem
14

As mentioned before I can make the Dockerfile above work by adding a tcp input that uses the same key. This will add the BouncyCastle provider which allows loading of the key.

FROM logstash:7.5.2
USER root
RUN yum install -y openssl
USER logstash
RUN openssl genrsa -out key.pem 2048
RUN openssl req -x509 -new -key key.pem -out cert.pem -days 365 -subj '/CN=localhost' -nodes
RUN printf 'input {\n'                                            > pipeline/logstash.conf
RUN printf '  tcp {\n'                                            >> pipeline/logstash.conf
RUN printf '    port => 1112\n'                                     >> pipeline/logstash.conf
RUN printf '    ssl_enable => true\n'                                        >> pipeline/logstash.conf
RUN printf '    ssl_cert => "/usr/share/logstash/cert.pem"\n'  >> pipeline/logstash.conf
RUN printf '    ssl_key => "/usr/share/logstash/key.pem"\n'          >> pipeline/logstash.conf
RUN printf '}\n'                                                 >> pipeline/logstash.conf
RUN printf '  beats {\n'                                            >> pipeline/logstash.conf
RUN printf '    port => 1111\n'                                     >> pipeline/logstash.conf
RUN printf '    ssl => true\n'                                        >> pipeline/logstash.conf
RUN printf '    ssl_certificate => "/usr/share/logstash/cert.pem"\n'  >> pipeline/logstash.conf
RUN printf '    ssl_key => "/usr/share/logstash/key.pem"\n'          >> pipeline/logstash.conf
RUN printf '}}\n'                                                 >> pipeline/logstash.conf
RUN printf 'output { stdout {} }'                               >> pipeline/logstash.conf
15

Hi @jsvd,

Thank you for all this tests. We already spent hours on this issue but I think we really need to dig more on this. The deployment workflow is quite painful and we inherit this big project from someone else without a handover. It's sometimes really complicated to do the reverse engineering / thinking process. We might be hitting a special case within our own platform. I'm going to spread the updates more widely and see how it goes (and in the worst case going for 7.5.2 on Logstash instead of 7.6.2).

I'll keep you posted in this thread.

In the meantime:

  1. I already have a piece of explanation for the working 7.0.1, 7.4.2, 7.5.2 versions (on the only cluster we tried: it can be a particular case): need to check all the sources put even if the config file inclue a SSL beats input, it also has a no-SSL one which seems to be used widely across the source clients (I need to check them all).

  2. However, and that's my main difficulty, I can't not find any explanation on the fact that manually converting the key in PKCS#8 is not working with 7.6.2 version. As you said testing this with a lower version is a non-sense because the key is not used unless a beat event actually comes in.

Obvious conclusion: at this point, and considering the fact that we are using git & Ansible to deploy configuration, we might face the same issue on all our clusters. What really upsets me is the fact that the only rational explanation I see at this point is: absolutely no source is using SSL at all. As this is a pre-requisite on the platform, as this is written in docs / arch schemas, and as this concerns hundreds of machines: I let you imagine how I feel right now :smiley:. Maybe we thought things were taken for granted when they weren't!

Again, I keep you informed here.

2 Likes
16

To sum it up, the beats input that ships with Logstash 7.6.2 will validate keys during pipeline start up, while older versions only checked when there's a new connection.

This should mean that current and older versions of logstash support the same keys, only that you get the error at different stages, and I'd posit that it's better as it is now, where validation happens sooner rather than later.

For folks landing here and who are able to push data to logstash < 7.6.2 but can't use that key in 7.6.2, if you're able to reproduce this with a similar throwaway key, please post the PEM encoded private key and the certificate so I'll be able to further debug this.

1 Like
18

Hello everyone,

I'm coming back to you. "Issue" is solved.
The root cause of our problem was wrong permissions on the manually PCKS#8 converted key. What make us struggle a lot was the inappropriate log message from Logstash plain text logs, see:

File does not contain valid private key: my_key.pem

By chance, I randomly decided to try the Logstash update on a host that has JSON logging enabled. Here, the message is totally different:

could not find key file: my_key.pem

So here are our conclusions:

  • Behaviours of our Logstash are consistent
  • Our Logstash never worked with SSL Beat pipelines
  • Our previous key was PCKS#1 file (with good permissions)
  • When I manually converted the PCKS#1 to PCKS#8 I was stupid enough not to check the permissions again. On this point Logstash logs were not helpful, may be it can be improved a bit.

Lessons learnt:

  • Biggest / more time-expensive issues are often stupid human mistakes :slight_smile:
  • JSON Logstash logs is preferable on the plain homolog

TL;DR: wrong permissions on the PCKS#8 converted key.

Thanks for reading,
Quentin

5 Likes
19

Big thanks to all for the information and testing!

I've created https://github.com/logstash-plugins/logstash-input-beats/pull/394 to distinguish between failure to read the files and invalid key materials.

1 Like
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.