Logstash connection failure to elasticsearch (deduplication pipeline)

I have created a pipeline to deduplicate logs in an index. I am facing issues while having logstash connect to the elasticsearch node.

IP hosting elasticsearch is 192.168.0.2 and has a self-signed certificate.

Following is the error, which I reckon is due to the self-signed certificate. How do solve it? (make it ignore the certificate authority not being in the root store.)

While I reckon it has no impact, logstash is running on a Raspebrry Pi running Raspbian OS (debian buster.)

Jun 16 01:01:33 HOSTNAME logstash[4912]: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:346:in `inputworker'
Jun 16 01:01:33 HOSTNAME logstash[4912]: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:337:in `block in start_input'
Jun 16 01:01:34 HOSTNAME logstash[4912]: ....[2020-06-16T01:01:34,263][ERROR][logstash.javapipeline    ][main][3ac418918ce70506b0ac53d00cf562268311b9742cc89d6f145f89775a16f11a] A plugin had an unrecoverable error. Will restart this plugin.
Jun 16 01:01:34 HOSTNAME logstash[4912]:   Pipeline_id:main
Jun 16 01:01:34 HOSTNAME logstash[4912]:   Plugin: <LogStash::Inputs::Elasticsearch password=><password>, hosts=>["192.168.0.2:9200"], query=>"{ \"sort\": [ \"_doc\" ] }", index=>"cloned-logstash-2020.01.07", id=>"3ac418918ce70506b0ac53d00cf562268311b9742cc89d6f145f89775a16f11a", ssl=>true, user=>"elastic", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_4365356d-45ad-4f71-a3ed-d2a0efb03ca0", enable_metric=>true, charset=>"UTF-8">, size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"]>
Jun 16 01:01:34 HOSTNAME logstash[4912]:   Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Jun 16 01:01:34 HOSTNAME logstash[4912]:   Exception: Manticore::ClientProtocolException
Jun 16 01:01:34 HOSTNAME logstash[4912]:   Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:321:in `search_request'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:269:in `do_run_slice'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:250:in `do_run'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:238:in `run'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:346:in `inputworker'
Jun 16 01:01:34 HOSTNAME logstash[4912]: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:337:in `block in start_input'
Jun 16 01:01:35 HOSTNAME logstash[4912]: .[2020-06-16T01:01:35,359][ERROR][logstash.javapipeline    ][main][3ac418918ce70506b0ac53d00cf562268311b9742cc89d6f145f89775a16f11a] A plugin had an unrecoverable error. Will restart this plugin.
Jun 16 01:01:35 HOSTNAME logstash[4912]:   Pipeline_id:main
Jun 16 01:01:35 HOSTNAME logstash[4912]:   Plugin: <LogStash::Inputs::Elasticsearch password=><password>, hosts=>["192.168.0.2:9200"], query=>"{ \"sort\": [ \"_doc\" ] }", index=>"cloned-logstash-2020.01.07", id=>"3ac418918ce70506b0ac53d00cf562268311b9742cc89d6f145f89775a16f11a", ssl=>true, user=>"elastic", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_4365356d-45ad-4f71-a3ed-d2a0efb03ca0", enable_metric=>true, charset=>"UTF-8">, size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"]>
Jun 16 01:01:35 HOSTNAME logstash[4912]:   Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Jun 16 01:01:35 HOSTNAME logstash[4912]:   Exception: Manticore::ClientProtocolException
Jun 16 01:01:35 HOSTNAME logstash[4912]:   Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:321:in `search_request'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:269:in `do_run_slice'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:250:in `do_run'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:238:in `run'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:346:in `inputworker'
Jun 16 01:01:35 HOSTNAME logstash[4912]: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:337:in `block in start_input'
Jun 16 01:01:36 HOSTNAME logstash[4912]: [2020-06-16T01:01:36,471][ERROR][logstash.javapipeline    ][main][3ac418918ce70506b0ac53d00cf562268311b9742cc89d6f145f89775a16f11a] A plugin had an unrecoverable error. Will restart this plugin.
Jun 16 01:01:36 HOSTNAME logstash[4912]:   Pipeline_id:main
Jun 16 01:01:36 HOSTNAME logstash[4912]:   Plugin: <LogStash::Inputs::Elasticsearch password=><password>, hosts=>["192.168.0.2:9200"], query=>"{ \"sort\": [ \"_doc\" ] }", index=>"cloned-logstash-2020.01.07", id=>"3ac418918ce70506b0ac53d00cf562268311b9742cc89d6f145f89775a16f11a", ssl=>true, user=>"elastic", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_4365356d-45ad-4f71-a3ed-d2a0efb03ca0", enable_metric=>true, charset=>"UTF-8">, size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"]>
Jun 16 01:01:36 HOSTNAME logstash[4912]:   Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Jun 16 01:01:36 HOSTNAME logstash[4912]:   Exception: Manticore::ClientProtocolException
Jun 16 01:01:36 HOSTNAME logstash[4912]:   Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:321:in `search_request'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:269:in `do_run_slice'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:250:in `do_run'
Jun 16 01:01:36 HOSTNAME logstash[4912]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.6.0/lib/logstash/inputs/elasticsearch.rb:238:in `run'

I am trying to install the certificate into Raspberry Pi using guide from this link:

https://www.raspberrypi.org/forums/viewtopic.php?t=204129

I will update here if it worked or not.

Thank you.

Even after trying steps suggested in following posts, I am unable to get rid of the error. I reckon the problem may be fixed by adding certificate in the local CA store. If anyone here knows about how to add it to Raspberry Pi, please do let me know.

Following threads did not help:

  1. https://raspberrypi.stackexchange.com/questions/76419/entrusted-certificates-installation
  2. https://www.raspberrypi.org/forums/viewtopic.php?t=204129

I get the following output which means the store did not get updated:

username@hostname:/usr/share/ca-certificates/local# sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.

Thank you.

@parthmaniar did you add valid certificate to logstash input/output.

Can you please add code snippet .

Thank you.

Its locally created certificate and hence the root CA isn't a trusted entity. Anyway I can add this to the store?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.