Enabling SSL with Elasticsearch cause logstash pipeline error

Elastic Stack Logstash
1

hi,

I am using 8.4.3, for both elastic and logstash.

We have enabled ssl with Elasticsearch. with the ssl our logstash fails to output data there, without ssl it works fine.

the output settings:

output{
elasticsearch {
		hosts => "elstichost:443"
		index => "%{[log-type]}-%{[a-type]}--%{+YYYY.MM}"
		ssl => "true"
		user => "****"
		password => "******"
		#ilm_enabled => false
        #manage_template => false
		cacert => "\logstash_cert.crt"
        keystore => "\logstash_key.key"
        ssl_certificate_verification => true
    }
}

The Error i am getting is:

[2022-11-16T14:10:43,383][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Java::JavaIo::IOException: toDerInputStream rejects tag type 45>, :backtrace=>["sun.security.util.DerValue.toDerInputStream(sun/security/util/DerValue.java:1155)", "sun.security.pkcs12.PKCS12KeySt
ore.engineLoad(sun/security/pkcs12/PKCS12KeyStore.java:2013)", "sun.security.util.KeyStoreDelegator.engineLoad(sun/security/util/KeyStoreDelegator.java:221)", "java.security.KeyStore.load(java/security/KeyStore.java:1473)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.refl
ect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMetho
d.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:427)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:294)", 
......


[2022-11-16T14:10:43,406][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-11-16T14:10:43,415][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-11-16T14:10:43,506][INFO ][logstash.runner          ] Logstash shut down.

Any help would be really appreciated.

Thanks

2

This does not looks like a valid path, have you tried to use the full path to both of those files?

1 Like
3

Thanks Leandro for your respinse actually in my config i used the full and the valid path here i just cropped it. I should have mentioned that. There was nothing invalid in this config.

Can you please see if there is anything else missing for the ssl configuration

I didn't find a full example of output with ssl enabled anywhere i picked these keys from different forums so i am.not sure if this is all we need to specify.

Thanks

4

Did you check the documentation ?

It explains what is every option in the output.

Normally in logstash you just need to configure de cacert option and point it to the self-signed CA used to sign the Elasticsearch certificates, this is also explained in the documentation.

Try to remove the keystore option and use the cacert option only.

Try this:

elasticsearch {
	hosts => "elstichost:443"
	index => "%{[log-type]}-%{[a-type]}--%{+YYYY.MM}"
	ssl => "true"
	user => "****"
	password => "******"
	#ilm_enabled => false
    #manage_template => false
	cacert => "path-to-the-ca-cert.crt"
    ssl_certificate_verification => true
}
5

If you are using backslash in the file paths try changing them to forward slash. I know in the file input that when javafication was done (V5.x?) the input started treating them as escapes.

6

Hi Thanks for your responses.

@Leandrojmp
I've already tried without the kaystore option that was giving me this error:

[2022-11-16T13:31:49,923][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2022-11-16T13:31:49,938][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash-agent:xxxxxx@elastichost:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=
>"Got response code '403' contacting Elasticsearch at URL 'https://elastichost:443/'"}

I am not sure why it indicates
"https://logstash-agent:xxxxxx@elastichost:443/"
in the error.
Is this an expected thing?

@Badger when i tried with the forward slash and without keystore option i get the same error

[2022-11-17T09:52:38,874][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash-agent:xxxxxx@elastichost:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError,
 :message=>"Got response code '403' contacting Elasticsearch at URL 'https://elastichost:443/'"}
 
when i try with keystore and forward slash i am back to the original error that i initially posted:

[2022-11-17T10:00:54,959][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Java::JavaIo::IOException: toDerInputStream rejects tag type 45>, :backtrace=>["sun.security.util.DerValue.toDerInputStream(sun/security/util/DerValue.java:1155)", "sun.security.pkcs12.P
KCS12KeyStore.engineLoad(sun/security/pkcs12/PKCS12KeyStore.java:2013)", "sun.security.util.KeyStoreDelegator.engineLoad(sun/security/util/KeyStoreDelegator.java:221)", "java.security.KeyStore.load(java/security/KeyStore.java:1473)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)"
, "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "o

elasticsearch {
        hosts => "elastichost:443"
        index => "%{[log-type]}-%{[a-type]}-%{[customer]}-debug-%{[log-timestamp-year]}.%{[log-timestamp-month]}.%{[log-timestamp-day]}"
		user => "****"
		password => "*****"
		ssl => "true"
        cacert => "C:/my/path/to/certificate/logstash_cert.crt"
        #keystore => "C:/my/path/to/logstash_key.key"
        ssl_certificate_verification => true		
      }

There is no difference in the error messages.

Is there anything else that i can have a look at?

Appreciate your help.

Thanks

7

Hi,

@Badger @leandrojmp
I couldn't find any solution for this. I have tried all the possible combination with the output settings as mentioned earlier but i am still unable to get it work anything you guys can indicate to help?

Thanks

Following is the error i get

WARNING: Illegal reflective access by org.jruby.javasupport.binding.ConstantField (file:/C:/Program%20Files/Elastic/logstash-8.5.2/vendor/jruby/lib/jruby.jar) to field sun.security.x509.X509CertImpl.SIG
WARNING: Please consider reporting this to the maintainers of org.jruby.javasupport.binding.ConstantField
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2022-11-28T13:37:31,351][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Java::JavaIo::IOException: toDerInputStream rejects tag type 45>, :backtrace=>["sun.security.util.DerValue.toDerInputStream(sun/security/util/DerValue.java:873)", "sun.security.pkcs12.PKCS12KeyStore.engineLoad(sun/security/pkcs12/PKCS12KeyStore.java:1997)", "sun.security.util.KeyStoreDelegator.engineLoad(sun/security/util/KeyStoreDelegator.java:222)", "java.security.KeyStore.load(java/security/KeyStore.java:1479)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:427)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:294)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.get_store(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:762)", "org.jruby.RubyKernel.tap(org/jruby/RubyKernel.java:1940)", "org.jruby.RubyKernel$INVOKER$s$0$0$tap.call(org/jruby/RubyKernel$INVOKER$s$0$0$tap.gen)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.get_store(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:760)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.setup_key_store(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:718)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.ssl_socket_factory_from_options(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:693)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.pool_builder(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:454)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.pool(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:462)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.manticore_minus_0_dot_9_dot_1_minus_java.lib.manticore.client.initialize(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:227)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.http_client.manticore_adapter.initialize(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_adapter(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:329)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_pool(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:345)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.http_client.initialize(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.create_http_client(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.build(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.plugin_mixins.elasticsearch.common.build_client(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:39)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_9_dot_3_minus_java.lib.logstash.outputs.elasticsearch.register(C:/Program Files/Elastic/logstash-8.5.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch.rb:296)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:572)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:348)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:275)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:131)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:117)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$register.call(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt$INVOKER$i$0$0$register.gen)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(C:/Program Files/Elastic/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:234)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(C:/Program Files/Elastic/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:233)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.logstash_minus_core.lib.logstash.java_pipeline.maybe_setup_out_plugins(C:/Program Files/Elastic/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:600)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(C:/Program Files/Elastic/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:246)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.logstash_minus_core.lib.logstash.java_pipeline.run(C:/Program Files/Elastic/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:191)", "C_3a_.Program_20_Files.Elastic.logstash_minus_8_dot_5_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start(C:/Program Files/Elastic/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:143)", "org.j

8

Have you simply tried to curl from logstash host to elasticsearch with the SSL?

Do you actually have elasticsearch running on 443 not 9200?

Start with

curl -k -v -u username https://eshost:port

9

@stephenb Thanks for your response.

Yes elastic is running on the host and i have tried following:

Yes i have tried the curl command i have curl installed on windows but i get the following error

Invoke-WebRequest : Parameter cannot be processed because the parameter name 'u' is ambiguous. Possible matches
include: -UseBasicParsing -Uri -UseDefaultCredentials -UserAgent.
At line:1 char:12

i tried without any parameter as well but nothing seems working

Although when i try via postman it does work for the same credentials.
I just provide
Method Put
the user/pwd and
the 'https://ealstichost/index-name1' and I was able to create an index

Also i can access the host via chrome, but when i try via logstash it get this error while providing only the user/pwd and no port

[2022-11-28T16:59:03,745][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash-agent:xxxxxx@elastichost:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elastichost:9200/][Manticore::ConnectTimeout] Connect to elastichost:9200 [ealstichost/] failed: connect timed out"}

I don't know what else should be tried here?

Thanks

10

http or https?

Can you share your actual logstash output again... it seems to be changing?

Also do you have more than 1 conf file in the conf.d directory? If so it will be concatenated together.

On curl not sure which curl you installed ... -u for user is a common option.

The just try

curl -k -v https://eshost:port

Nothing that you have showed shows that there is network connectivity between the logstash server and the elasticsearch server.

Also to test you could turn ssl verification off to test

   elasticsearch {
     hosts => "https://elastichost:9200"
     index => "%{[log-type]}-%{[a-type]}-%{[customer]}-debug-%{[log-timestamp-year]}.%{[log-timestamp-month]}.%{[log-timestamp-day]}"
     user => "****"
     password => "*****"
     ssl_certificate_verification => false		
  }
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.