Logstash 7.2 exception when enabling TLS and X-pack (error with elastic output)

Elastic Stack Logstash
1

Hello.

After an upgrade to Logstash 7.2 my config using Ruby code fails to start.

The Ruby code is basic inline using "ruby { code => '' } "

Error message below. Thank you for the help.

Here is the giant error, it is too big for a single post so I will break it up into two.

[2019-07-21T10:52:06,171][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-07-21T10:52:06,185][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-21T10:52:11,234][ERROR][logstash.javapipeline ] java.lib.manticore.client.pool_builder(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:397)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$pool_builder$0$VARARGS(opt/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.pool(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:405)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.initialize(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:209)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.manticore_adapter.initialize(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_adapter(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:282)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.RUBY$method$build_adapter$0$VARARGS(opt/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_pool(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:286)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.RUBY$method$build_pool$0$VARARGS(opt/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.initialize(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:64)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.create_http_client(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:103)",

2

Continuation of error message from initial post:

"opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$create_http_client$0$VARARGS(opt/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.build(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:99)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$build$0$VARARGS(opt/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.build_client(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb:238)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.RUBY$method$build_client$0$VARARGS(opt/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs//opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb)", "opt.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.register(/opt/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:25)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:548)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:354)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:246)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:106)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:91)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$register.call(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt$INVOKER$i$0$0$register.gen)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:192)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$VARARGS(opt/share/logstash/logstash_minus_core/lib/logstash//opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.maybe_setup_out_plugins(/opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:462)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$maybe_setup_out_plugins$0$VARARGS(opt/share/logstash/logstash_minus_core/lib/logstash//opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:204)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$VARARGS(opt/share/logstash/logstash_minus_core/lib/logstash//opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$VARARGS(opt/share/logstash/logstash_minus_core/lib/logstash//opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "opt.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/opt/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:105)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"], :thread=>"#<Thread:0x2ab85d88 run>"}
[2019-07-21T10:52:11,258][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2019-07-21T10:52:11,543][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-07-21T10:52:16,633][INFO ][logstash.runner ] Logstash shut down.

3

The error appears to be in an elasticsearch output, not in a ruby filter. Can you show us your configuration? Did you upgrade elasticsearch to 7.2 along with logstash?

4

Thanks, that helps, it's probably the TLS and auth setup.

Yes, Elasticsearch was upgraded to 7.2 prior to the Logstash upgrade. Logstash remained at 6.8 and continued to operate.

The major change was enabling x-pack and TLS on the ES cluster. Logstash was then upgraded to 7.2 and modified to use TLS and authentication.

I've followed the guides for creating users. The logstash user has all relevant privileges. The logstash TLS config is somewhat unclear compared to the config for enabling cluster TLS. Maybe this is where the problem is.

The ES output config is:

elasticsearch {
hosts => ["https://10.1.1.1:9200"]
codec => "plain"
sniffing => false
index => "logfiles-%{+YYYY.MM.dd}"
manage_template => false
template_name => "logfiles_template"
template_overwrite => false
ssl => true
ssl_certificate_verification => false
cacert => "/etc/logstash/certs/elastic-ca.pem"
user => "user"
password => "password"
}

5

The place where the exception occurs is certainly TLS related. You might want to fix the title of the thread to be 'Logstash 7.2 exception when enabling TLS and X-pack'. I do not run elasticsearch so I cannot help further.

6

Thanks, I will change the subject.

You helped me see the problem differently where I could solve it after slight config modifications. The CA cert wasn't in the correct format (.pem vs .p12)

Now there's a different error about "ressurrect connection to dead ES instance" relating to 'response code 403' which I think may be related to ES user permissions.

7

This issue is resolved. The user_role was lacking specifics from the Elastic documentation.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.