Hi Team,
I have deployed a 2-node ELK stack cluster where I can successfully login to kibana instance with my superuser credentials.
All of the ELK components (running on same Ubuntu VM) including metricbeats is running version 7.17.1.
I have xpath security enabled on both the nodes which can be seen as follows -
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: cert/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: cert/elastic-stack-ca.p12
The problem is while I am trying to stop/start/restart logstash service, it just hangs in there with no output at all.
Verified the logstash-test.conf file under /etc/logstash/conf.d/ with command "bin/logstash -f logstash-test.conf --config.test_and_exit" and it looked fine.
[WARN ] 2022-03-11 05:32:53.291 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2022-03-11 05:32:53.386 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
Looking at the /var/log/logstash/logstash-plain.log as well as running command "bin/logstash -f logstash-test.conf --config.reload.automatic" results in same log events which are as follows -
[2022-03-11T04:09:40,940][DEBUG][logstash.outputs.elasticsearch][syslog-test] Running health check to see if an ES connection is working {:url=>"http://<ip_masked>:9200/", :path=>"/"}
[2022-03-11T04:09:40,948][WARN ][logstash.outputs.elasticsearch][syslog-test] Failed to perform request {:message=>"Unsupported or unrecognized SSL message",
:exception=>Manticore::UnknownException, :cause=>javax.net.ssl.SSLException: Unsupported or unrecognized SSL message, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/response.rb:36:in `block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:73:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:233:in `health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:238:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:215:in `block in start_resurrectionist'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:136:in `until_stopped'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:214:in `block in start_resurrectionist'"]}
[2022-03-11T04:09:40,949][DEBUG][logstash.outputs.elasticsearch.httpclient.manticoreadapter][syslog-test]
Surprisingly, I am able to curl to http://<elasticsearch_ip_masked>:9200 without user credentials on the browser, however, the same is getting failed with/without user credentials via CLI (w/ -u username:password).
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
I have already spent a lot of time on troubleshooting and reviewing Configuring Security in Logstash | Logstash Reference [7.17] | Elastic, however, couldn't really make any progress.
Any inputs from your side in order to debug or isolate the issue would be highly appreciated.
Thanks,
Rahul