Logstash TCP Input with SSL failing with non descript error

Elastic Stack Logstash
1

I'm trying to set up a TCP Input with an SSL certificate, but no matter how I configure it, i keep getting a non descript error. This is running on Logstash 7.9.1 OSS.

My config is the following:

input {

        tcp {
                type  => 'syslog'
                port => 8378
                ssl_enable => false
                ssl_cert => '/usr/share/logstash/config/logstash-crt.pem'
                ssl_key =>  '/usr/share/logstash/config/logstash-key.pem'
                #ssl_certificate_authorities => ['10.10.10.2', '10.10.10.3']
                ssl_key_passphrase => pwd
                #ssl_verify => false
                dns_reverse_lookup_enabled => false
        } #end TCP

} #end inputs

output {
        pipeline {
                send_to => ["filter-debug"]
        }
}

The pipeline works just fine with "ssl_enable" set to false. As soon as it is set to "true" i get the error listed below. It doesnt matter if i have all, some or none of the other configs set. If i add "ssl_enable" to any pipeline, it fails with the same unfortunately rather useless error message.

Logstash log:

[2021-11-03T10:17:29,314][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>"input-debug"}
[2021-11-03T10:17:30,056][INFO ][logstash.javapipeline    ][input-debug] Starting pipeline {:pipeline_id=>"input-debug", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/inputs/input-debug.logstash.conf"], :thread=>"#<Thread:0x38d377c4 run>"}
[2021-11-03T10:17:30,064][INFO ][logstash.javapipeline    ][input-debug] Pipeline Java execution initialization time {"seconds"=>0.01}
[2021-11-03T10:17:30,191][ERROR][logstash.agent           ] Failed to execute action {:id=>:"input-debug", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Reload<input-debug>, action_result: false", :backtrace=>nil}
2

Change ssl_certificate_authorities to file and ssl_verify => false.
Certificates will work:

  • certificate_authorities - Should be path to root cert in a file. Validate client certificate or certificate chain against these authorities. You can define multiple files or paths. All the certificates will be read and added to the trust store.
    From Filebeat: Configures Filebeat to trust any certificates signed by the specified CA. If certificate_authorities is empty or not set, the trusted certificate authorities of the host system are used.
  • ssl_verify => false, uncomment until you are sure this is the last issue.Actually means check do I trust certificate, does the same CA issued certificates
  • Certificate must be issued and used to hostname or IP. It's possible both if you specify as alternative name. Anyway, how is issued the certificate, that should be in connection to Logstash.
    Use openssl or curl to test correct SSL session.
1 Like
3

Thanks for replying - so even if i run it with bare minimum:

input {

        tcp {
                port => 8378
                ssl_enable => true
                ssl_verify => false
        } #end TCP

} #end inputs

output {
        pipeline {
                send_to => ["filter-debug"]
        }
}

I still get the same error. Something suggests a bug to me. I do not see any requirements in the documentation. Have you gotten it to work?

4

I have also tested the certificates, they work. They are identical in this test setup to the ones used by my Kibana instance, as they are on the same test environment and this is just an initial test.

5

Try this:


input {
        tcp {
                type  => 'syslog'
                port => 8378
                ssl_enable => true
                ssl_cert => '/usr/share/logstash/config/logstash-crt.pem'
                ssl_key =>  '/usr/share/logstash/config/logstash-key.pem'
                ssl_certificate_authorities => '/usr/share/logstash/config/ca.crt'
                # remove pass from cert, ssl_key_passphrase => pwd
                ssl_verify => false
                dns_reverse_lookup_enabled => false
        } #end TCP

}

Certificates must be in PEM-Base64 format. Also try to put cert files in the /etc/logstash directory.
On other-client side also use hostname not IP.

6

Try enabling log.level TRACE. There are logstash versions (and I think this was around 7.9) where some certificate key error messages were logged at the wrong level.

1 Like
7

So i've tried enabling Trace - it took a while to sift through as there are thousands of messages logged.

Here is the snippet from where the pipeline is created:

[2021-11-04T09:41:33,508][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
[2021-11-04T09:41:33,508][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:input-debug}
[2021-11-04T09:41:33,514][DEBUG][org.logstash.secret.store.SecretStoreFactory] Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[2021-11-04T09:41:34,070][DEBUG][logstash.codecs.line     ] config LogStash::Codecs::Line/@id = "line_9b3be852-92d7-4425-89e6-884be3b89216"
[2021-11-04T09:41:34,070][DEBUG][logstash.codecs.line     ] config LogStash::Codecs::Line/@enable_metric = true
[2021-11-04T09:41:34,070][DEBUG][logstash.codecs.line     ] config LogStash::Codecs::Line/@charset = "UTF-8"
[2021-11-04T09:41:34,071][DEBUG][logstash.codecs.line     ] config LogStash::Codecs::Line/@delimiter = "\n"
[2021-11-04T09:41:34,071][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@port = 8378
[2021-11-04T09:41:34,071][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@ssl_verify = false
[2021-11-04T09:41:34,071][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@ssl_enable = true
[2021-11-04T09:41:34,071][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@id = "3a6289309f2a750e47cb25be20208f13dba152b8e1beaf6c352ba0de192b8269"
[2021-11-04T09:41:34,071][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@type = "syslog"
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@ssl_certificate_authorities = ["/usr/share/logstash/config/ca.pem"]
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@enable_metric = true
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@codec = <LogStash::Codecs::Line id=>"line_9b3be852-92d7-4425-89e6-884be3b89216", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@add_field = {}
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@host = "0.0.0.0"
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@mode = "server"
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@proxy_protocol = false
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@ssl_extra_chain_certs = []
[2021-11-04T09:41:34,072][DEBUG][logstash.inputs.tcp      ] config LogStash::Inputs::Tcp/@tcp_keep_alive = false
[2021-11-04T09:41:34,074][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@id = "plain_8cf8127b-dccf-4993-9e66-dce004c637f4"
[2021-11-04T09:41:34,074][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@enable_metric = true
[2021-11-04T09:41:34,074][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2021-11-04T09:41:34,074][DEBUG][logstash.plugins.builtin.pipeline.output] config LogStash::Plugins::Builtin::Pipeline::Output/@send_to = ["filter-debug"]
[2021-11-04T09:41:34,075][DEBUG][logstash.plugins.builtin.pipeline.output] config LogStash::Plugins::Builtin::Pipeline::Output/@id = "f9f5d8275811f04ea2ca0dc82d572552e026e52cf328a95d4fae7c8146d33483"
[2021-11-04T09:41:34,075][DEBUG][logstash.plugins.builtin.pipeline.output] config LogStash::Plugins::Builtin::Pipeline::Output/@enable_metric = true
[2021-11-04T09:41:34,075][DEBUG][logstash.plugins.builtin.pipeline.output] config LogStash::Plugins::Builtin::Pipeline::Output/@codec = <LogStash::Codecs::Plain id=>"plain_8cf8127b-dccf-4993-9e66-dce004c637f4", enable_metric=>true, charset=>"UTF-8">
[2021-11-04T09:41:34,075][DEBUG][logstash.plugins.builtin.pipeline.output] config LogStash::Plugins::Builtin::Pipeline::Output/@workers = 1
[2021-11-04T09:41:34,075][DEBUG][logstash.plugins.builtin.pipeline.output] config LogStash::Plugins::Builtin::Pipeline::Output/@ensure_delivery = true
[2021-11-04T09:41:34,076][DEBUG][org.logstash.ackedqueue.QueueUpgrade] PQ version file with correct version information (v2) found.
[2021-11-04T09:41:34,076][DEBUG][org.logstash.ackedqueue.Queue] opening head page: 7, in: /usr/share/logstash/data/queue/input-debug, with checkpoint: pageNum=7, firstUnackedPageNum=7, firstUnackedSeqNum=0, minSeqNum=0, elementCount=0, isFullyAcked=no
[2021-11-04T09:41:34,079][DEBUG][org.logstash.ackedqueue.io.MmapPageIOV2] PageIO recovery element index:0, readNextElement exception: Element invalid length
[2021-11-04T09:41:34,082][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"input-debug"}
[2021-11-04T09:41:34,084][INFO ][logstash.javapipeline    ][input-debug] Starting pipeline {:pipeline_id=>"input-debug", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/inputs/input-debug.logstash.conf"], :thread=>"#<Thread:0xbdcfd06 run>"}
[2021-11-04T09:41:34,086][DEBUG][org.logstash.config.ir.CompiledPipeline][input-debug] Compiled output
 P[output-pipeline{"send_to"=>["filter-debug"]}|[file]/usr/share/logstash/pipeline/inputs/input-debug.logstash.conf:20:9:```
pipeline {
                send_to => ["filter-debug"]
        }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@3fdc2b9c
[2021-11-04T09:41:34,087][INFO ][logstash.javapipeline    ][input-debug] Pipeline Java execution initialization time {"seconds"=>0.0}
[2021-11-04T09:41:34,088][DEBUG][logstash.javapipeline    ][input-debug] Shutdown waiting for worker thread {:pipeline_id=>"input-debug", :thread=>"#<Thread:0x556758a0 run>"}
[2021-11-04T09:41:34,188][DEBUG][logstash.plugins.builtin.pipeline.output][input-debug] Closing {:plugin=>"LogStash::Plugins::Builtin::Pipeline::Output"}
[2021-11-04T09:41:34,188][DEBUG][logstash.pluginmetadata  ][input-debug] Removing metadata for plugin f9f5d8275811f04ea2ca0dc82d572552e026e52cf328a95d4fae7c8146d33483
[2021-11-04T09:41:34,188][DEBUG][logstash.javapipeline    ][input-debug] Pipeline terminated by worker error {:pipeline_id=>"input-debug", :exception=>java.lang.NullPointerException, :backtrace=>["java.io.FileInputStream.<init>(java/io/FileInputStream.java:149)", "jdk.internal.reflect.GeneratedConstructorAccessor94.newInstance(jdk/internal/reflect/GeneratedConstructorAccessor94)", "jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:490)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:285)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:918)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.compat_ssl_options.fetch_certificates_from_file(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp/compat_ssl_options.rb:132)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.compat_ssl_options.toSslContext(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp/compat_ssl_options.rb:81)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.get_ssl_context(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb:369)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.register(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.RUBY$method$register$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java/lib/logstash/inputs//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:226)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:225)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:359)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_inputs$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:834)"], "pipeline.sources"=>["/usr/share/logstash/pipeline/inputs/input-debug.logstash.conf"], :thread=>"#<Thread:0xbdcfd06 run>"}
[2021-11-04T09:41:34,194][ERROR][logstash.agent           ] Failed to execute action {:id=>:"input-debug", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<input-debug>, action_result: false", :backtrace=>nil}
[2021-11-04T09:41:34,194][TRACE][logstash.agent           ] Converge results {:success=>false, :failed_actions=>["id: input-debug, action_type: LogStash::PipelineAction::Create, message: Could not execute action: PipelineAction::Create<input-debug>, action_result: false"], :successful_actions=>[]}
8

Im not getting much from the stacktrace unfortunately. My next step is to try using a newer version to see if it is a bug for that version. If anyone can read that stacktrace and get something out of it, please pitch in.

9

What is on a client side which sends data to LS? Is there everything set?

Why are you sending to pipeline output? Can you use stdout {codec => rubydebug}

10

Nothing is on the client side. This error happens before the pipeline is initiated.

We use a pipeline to pipeline structure because we have a lot of pipelines and need to be able to route between them. I get the same error if i pipe to Stdout. The error happens when the pipeline is being initiated, not while it is running.

11 
[2021-11-04T09:41:34,188][DEBUG][logstash.javapipeline    ][input-debug] Pipeline terminated by worker error {
:pipeline_id=>"input-debug", 
:exception=>java.lang.NullPointerException, 
:backtrace=>[
"java.io.FileInputStream.<init>(java/io/FileInputStream.java:149)", 
....
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.compat_ssl_options.fetch_certificates_from_file(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp/compat_ssl_options.rb:132)", 
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.compat_ssl_options.toSslContext(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp/compat_ssl_options.rb:81)",

compat_ssl_options.rb no longer exists in the current codebase (it underwent javafication), but looking at an old fork of it that appears to be reading the ssl_cert option. I have no idea how that could end up being nil.

1 Like
12

I have now checked with:
Logstash 7.15.1 OSS in Docker
Logstash 7.15.1 In Docker
Logstash 7.15.1 installed with .deb on host

I get the same result.

13

I found the issue.

During my testing i found that 2 things can cause the null pointer we saw earlier:

  • If the certificate is missing
  • If the certificate has the wrong permission (ownership).

In my case it was the latter, i had created the certificates as root:root on accident. It worked like a charm when chowning to 1000:1000.

The reason i did not think of this , is that this should not cause a nullpointer. It should cause a permission error, like all other files in Logstash. I would still consider this a bug for that reason, but now we know.

Bonus:
For anyone else reading this, you can verify that your ssl pipeline is working by establishing a connection using s_client from openssh as such:

openssl s_client -connect localhost:8517 -cert /var/certs/somecert.crt -key /var/certs/somekey.key
1 Like
14

Interesting. java will throw the NPE if it cannot expand the file name into a path (for the reasons you gave). Ever since the javafication of the pipeline there are a lot more cases where errors crash logstash instead of being caught and handled.

1 Like
15

It sounds to me that what we need here is to wrap the entire codebase in a Try Catch! (That was a joke)

Thank you for your help Badger, I've marked the thread as solved and hope it will help others.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.