Issue with logstash (1:7.17.1-1) service on Ubuntu 18.04.3 LTS

Hi Team,

I have deployed a 2-node ELK stack cluster where I can successfully login to kibana instance with my superuser credentials.

All of the ELK components (running on same Ubuntu VM) including metricbeats is running version 7.17.1.

I have xpath security enabled on both the nodes which can be seen as follows -

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: cert/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: cert/elastic-stack-ca.p12

The problem is while I am trying to stop/start/restart logstash service, it just hangs in there with no output at all.

Verified the logstash-test.conf file under /etc/logstash/conf.d/ with command "bin/logstash -f logstash-test.conf --config.test_and_exit" and it looked fine.

[WARN ] 2022-03-11 05:32:53.291 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2022-03-11 05:32:53.386 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Looking at the /var/log/logstash/logstash-plain.log as well as running command "bin/logstash -f logstash-test.conf --config.reload.automatic" results in same log events which are as follows -


[2022-03-11T04:09:40,940][DEBUG][logstash.outputs.elasticsearch][syslog-test] Running health check to see if an ES connection is working {:url=>"http://<ip_masked>:9200/", :path=>"/"}
[2022-03-11T04:09:40,948][WARN ][logstash.outputs.elasticsearch][syslog-test] Failed to perform request {:message=>"Unsupported or unrecognized SSL message", 
:exception=>Manticore::UnknownException, :cause=>javax.net.ssl.SSLException: Unsupported or unrecognized SSL message, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/response.rb:36:in `block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:73:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:233:in `health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:238:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:215:in `block in start_resurrectionist'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:136:in `until_stopped'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:214:in `block in start_resurrectionist'"]}
[2022-03-11T04:09:40,949][DEBUG][logstash.outputs.elasticsearch.httpclient.manticoreadapter][syslog-test]

Surprisingly, I am able to curl to http://<elasticsearch_ip_masked>:9200 without user credentials on the browser, however, the same is getting failed with/without user credentials via CLI (w/ -u username:password).

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

I have already spent a lot of time on troubleshooting and reviewing Configuring Security in Logstash | Logstash Reference [7.17] | Elastic, however, couldn't really make any progress.

Any inputs from your side in order to debug or isolate the issue would be highly appreciated.

Thanks,
Rahul

http? So you do not have SSL configured?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.