Elasticsearch output plugin ssl verification

Hi elastic team
I've configured a pipeline containing a file input, some filters and elasticsearch output. the target es server is secured by ssl and the certificates are generated using the the way explained in docs. note that the es and logstash versions are 6.8
the problem I facing is that when I use the elastic-stack-ca.p12 file for 'cacert' param in elasticsearch output, the following error is generated in logstash (signed fields invalid):

[2023-01-01T16:41:03,368][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x24c5b7c5>", :error=>"signed fields invalid", :thread=>"#<Thread:0x6b3a135e run>"}
[2023-01-01T16:41:03,369][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>java.security.cert.CertificateParsingException: signed fields invalid, :backtrace=>["sun.security.x509.X509CertImpl.parse(sun/security/x509/X509CertImpl.java:1829)", "sun.security.x509.X509CertImpl.<init>(sun/security/x509/X509CertImpl.java:194)", "sun.security.provider.X509Factory.parseX509orPKCS7Cert(sun/security/provider/X509Factory.java:476)", "sun.security.provider.X509Factory.engineGenerateCertificates(sun/security/provider/X509Factory.java:361)", "java.security.cert.CertificateFactory.generateCertificates(java/security/cert/CertificateFactory.java:478)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:455)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:316)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_7_dot_0_minus_java.lib.manticore.client.setup_trust_store(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/client.rb:635)", "org.jruby.RubyIO.ensureYieldClose(org/jruby/RubyIO.java:1163)", "org.jruby.RubyIO.open(org/jruby/RubyIO.java:1157)", "org.jruby.RubyKernel.open(org/jruby/RubyKernel.java:320)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_7_dot_0_minus_java.lib.manticore.client.setup_trust_store(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/client.rb:634)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_7_dot_0_minus_java.lib.manticore.client.ssl_socket_factory_from_options(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/client.rb:622)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_7_dot_0_minus_java.lib.manticore.client.pool_builder(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/client.rb:398)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_7_dot_0_minus_java.lib.manticore.client.pool(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/client.rb:406)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_7_dot_0_minus_java.lib.manticore.client.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/client.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.manticore_adapter.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_adapter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:282)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_pool(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:286)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:64)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.create_http_client(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:103)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.build(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:99)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.build_client(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch.rb:238)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.RUBY$method$build_client$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java/lib/logstash/outputs//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.register(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:25)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:548)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:354)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:246)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:106)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:91)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.register_plugin(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$register_plugin$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$register_plugins$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.maybe_setup_out_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:611)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$maybe_setup_out_plugins$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:280)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$start_workers$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$run$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:829)"], :thread=>"#<Thread:0x6b3a135e run>"}
[2023-01-01T16:41:03,380][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

I'm not sure if it's the true way of using ca file to validate ssl certificate among elastic products but this was the way I understood from the docs.

the pipeline structure is like the below:

input {
  file {
  ...
  }
}
filter {
...
}
output {
  elasticsearch {
    hosts => [ "https://*.*.*.*:9200" ]
    cacert => "/etc/logstash/file_to_ES_input/elastic-stack-ca.p12"
    user => elastic
    password => "***"
    index => "sample-files"
  }
}

I would appreciate any helps for this issue.

From the documentation of the elasticsearch output you have:

The .cer or .pem file to validate the server’s certificate

You need to convert the .p12 to .pem, check this link for some way to do that.

1 Like

thanks Leandro. this solved the parsing error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.