Block in initialize'", block in healthcheck!' [LOGSTASH]

Elastic Stack Logstash
1

I installed brand new logstash and I got the same error. logstash.conf is working config and systax is ok but I got pipeline error. systemctl start logstash - is running but on logstash-plain.log there are some errors as below regarding pipeline I have only single config file which works fine already in prod. Anyone could help me? I am suffering about this issue.

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-4192/jruby14578939057917417933jopenssl.jar) to field java.security.MessageDigest.provider

WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[INFO ] 2021-07-12 15:56:56.516 [main] runner - Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}

[WARN ] 2021-07-12 15:56:57.043 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified

[INFO ] 2021-07-12 15:56:59.245 [Converge PipelineAction::Create<main>] Reflections - Reflections took 26 ms to scan 1 urls, producing 23 keys and 47 values

[WARN ] 2021-07-12 15:56:59.524 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.

[WARN ] 2021-07-12 15:56:59.553 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.

[INFO ] 2021-07-12 15:56:59.869 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@elklogstash01:9200/]}}

[ERROR] 2021-07-12 15:57:00.109 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<Manticore::UnknownException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in `block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:74:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:261:in `health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:270:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:266:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:382:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:82:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:76:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:302:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:64:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:105:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:101:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch.rb:307:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:23:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/beats.conf"], :thread=>"#<Thread:0xcaa0ea9 run>"}

[INFO ] 2021-07-12 15:57:00.111 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}

[ERROR] 2021-07-12 15:57:00.127 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[INFO ] 2021-07-12 15:57:00.272 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

[INFO ] 2021-07-12 15:57:05.314 [LogStash::Runner] runner - Logstash shut down.
2

Reading the trust store (either the default or the one you specified) did not result in reading any certificates. That could mean it is empty or missing, or you do not have permissions to read it. More discussion here.

3

Hi Badger,

Thank you for your response. it seems like we are on the right track. is it about the certificates which is configured in logstash.conf or do you mean another certificate which java might be using which i dont know but there is a certs directory and inside there are the certificates. I will check directory permissions as recursive.

4

The elasticsearch output is using an https URI. It needs a CA cert to be able to validate the certificate that elasticsearch is using. You may have supplied that using the cacert option. If not then I expect it will use the JVM's default truststore.

Whereever it is looking, it is not finding any CA certs there.

5

on the logstash.conf at the output section there is a line which shows cacert => /etc/logstash/certs/ca.crt normally it used to work fine until i add a new logstash config and restart the logstash. I reinstalled the logstash VM and reinstalled everything from the scratch and copy the certs from previous machine to the new VM however the error stayed there. I do not understand how i brake up a running environment by the way. currently i do not have access to this environment but i will have an access in the morning. I will check read\write permissions for certs directory chmod -R 777 then if it still does not work what should i do next.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.