Logstash beats input cant start with ssl enabled

Elastic Stack Logstash
1

I would like to make secure connection between fielbeat and logstash. Both version 7.8.1.

input {
  beats {
    port => 5000
    ssl => true
    ssl_certificate => "/ssl/instance.crt"
    ssl_key => "/ssl/instance.key"
  }
}

This is my input configured. instance.crt and instance.key match - checked. When I start logstash, I receive only one eerror line:

2020-09-23T09:16:19.007068685Z [2020-09-23T09:16:19,006][INFO ][logstash.inputs.beats    ][pipe2] Beats inputs: Starting input listener {:address=>"0.0.0.0:5000"}
2020-09-23T09:16:19.365623492Z [2020-09-23T09:16:19,364][ERROR][logstash.agent           ] Failed to execute action {:id=>:pipe2, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<pipe2>, action_result: false", :backtrace=>nil}
2020-09-23T09:16:19.716687784Z [2020-09-23T09:16:19,716][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
2020-09-23T09:16:21.605038810Z [2020-09-23T09:16:21,604][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
2020-09-23T09:16:21.667200487Z [2020-09-23T09:16:21,666][INFO ][logstash.runner          ] Logstash shut down.

I dont know how to fix it, because I dont know where is the problem. According to documentation, I configured everything right. Why I cant start logstash? Do I missing somethig?

I run everything locally right now. Version of logstash: 7.8.1. Directory /ssl/ with all files exists. I am running logstash in docker.

2

If you increase log.level to debug do you get any relavent messages?

4

Hi, thank you, I thought info about wrong key or cert would not be in debug.This is the output:

[2020-09-23T14:02:07,961][INFO ][logstash.inputs.beats    ][pipe2] Beats inputs: Starting input listener {:address=>"0.0.0.0:5001"}
[2020-09-23T14:02:07,962][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[2020-09-23T14:02:07,962][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[2020-09-23T14:02:07,962][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[2020-09-23T14:02:07,963][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2020-09-23T14:02:07,963][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
[2020-09-23T14:02:07,963][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
[2020-09-23T14:02:07,963][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2020-09-23T14:02:07,963][DEBUG][org.logstash.netty.SslContextBuilder][pipe2] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
[2020-09-23T14:02:07,971][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x1a662e0a run>"}
[2020-09-23T14:02:08,059][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x45ace3be dead>"}
[2020-09-23T14:02:08,059][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x42326eb8 dead>"}
[2020-09-23T14:02:08,059][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0xdbf3608 dead>"}
[2020-09-23T14:02:08,059][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x4de86e28 dead>"}
[2020-09-23T14:02:08,059][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x727ea9ea run>"}
[2020-09-23T14:02:08,060][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x14da5447 run>"}
[2020-09-23T14:02:08,061][DEBUG][logstash.javapipeline    ][pipe2] Shutdown waiting for worker thread {:pipeline_id=>"pipe2", :thread=>"#<Thread:0x3bc36e65 run>"}
[2020-09-23T14:02:08,061][DEBUG][logstash.outputs.stdout  ][pipe2] Closing {:plugin=>"LogStash::Outputs::Stdout"}
[2020-09-23T14:02:08,061][DEBUG][logstash.pluginmetadata  ][pipe2] Removing metadata for plugin bd06649debcba9d1a2b0d1c7a585143835583bba947482143d11e1509a12e4f0
[2020-09-23T14:02:08,062][DEBUG][logstash.javapipeline    ][pipe2] Pipeline terminated by worker error {:pipeline_id=>"pipe2", :exception=>java.lang.IllegalArgumentException: File does not contain valid private key: /ssl/instance.key, :backtrace=>["io.netty.handler.ssl.SslContextBuilder.keyManager(io/netty/handler/ssl/SslContextBuilder.java:350)", "io.netty.handler.ssl.SslContextBuilder.forServer(io/netty/handler/ssl/SslContextBuilder.java:107)", "org.logstash.netty.SslContextBuilder.buildContext(org/logstash/netty/SslContextBuilder.java:145)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.create_server(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats.rb:180)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.RUBY$method$create_server$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java/lib/logstash/inputs//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.register(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats.rb:156)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.RUBY$method$register$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java/lib/logstash/inputs//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:226)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:225)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:355)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_inputs$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:305)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:834)"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x688f83d7 run>"}
[2020-09-23T14:02:08,068][ERROR][logstash.agent           ] Failed to execute action {:id=>:pipe2, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Reload<pipe2>, action_result: false", :backtrace=>nil}

It looks like problem is with private key:
File does not contain valid private key: /ssl/instance.key

But I checked it first with:

openssl pkey -in instance.key -pubout -outform pem | sha256sum 
239aab42fac5486d51bff39807d8c22dfe89fc3a525ae4ec40e60cc78297f2fe  -
openssl x509 -in instance.crt -pubkey -noout -outform pem | sha256sum
239aab42fac5486d51bff39807d8c22dfe89fc3a525ae4ec40e60cc78297f2fe  -

SHA is the same, so am I checking it wrong way? How should I checked it?

5

SOLUTION:

So the problem was invalid private key. I set up debug log level for logstash in logstash.yml:

log.level: debug

Then, in all debug messages, there was message about invalid private key. At first, this should be mentioned, according to me, in INFO logs. The certificate, as I mentioned above, is valid, but the format of certificate was wrong. So from documentation of input beats:

SSL key to use. NOTE: This key need to be in the PKCS8 format, you can convert it with OpenSSL for more information.

So I change format with:

openssl pkcs8 -topk8 -nocrypt -in instance.key -out instance.key.pkcs8

And change my configuration of input beats:

input {
  beats {
    port => 5000
    ssl => true
    ssl_certificate => "/ssl/instance.crt"
    ssl_key => "/ssl/instance.key.pkcs8"
  }
}

Than it starts to work. Everywhere in cluster (filebeat, elasticsearch) was this cert valid, but for logstash, there must be different format.