Logstash clustering issue

Im having Elasticsearch clustering and my output.conf in logstash, looks like below

hosts => ["node1:9200","node2:9200","node3:9200"]

So through java, Im updating output.conf and starting logstash. Now in case my elasticsearch cluster nodes takes some time to come up, logstash throws an error

[2020-10-12T17:52:25,998][ERROR][logstash.outputs.elasticsearch][events] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://164.99.185.202:9200/][Manticore::SocketException] Connection refused (Connection refused)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :backtrace=>["logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:293:in perform_request_to_url'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in block in perform_request'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in with_connection'", logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in block in Pool'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in get'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:378:in get_xpack_info'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:57:in ilm_ready?'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:28:in ilm_in_use?'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:14:in install_template'",
logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:130:in install_template'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:51:in block in setup_after_successful_connection'"]}
[2020-10-12T17:52:26,037][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Is it stopped as elasticsearch nodes are down(one node is still up)?? I thought even though elasticsearch nodes are down, logstash will try to establish communication . It should not get shut down. Am I correct?

Try enabling log.level debug to see if you get a more informative error message.

Sure. But whats the expectation?? Even though all elasticsearch nodes are down, logstash should not stop. It should try to establish communication.Is that right???

The output thinks it has successfully connected to elasticsearch, so it is trying to do template management. It could be a race condition where if elasticsearch goes down between the test for a connection and the attempt to use it then the failure is not handled. I do not know.

Generally I would expect an elasticsearch output to be OK even if all hosts are down when it is registered.

This is the only Fatal error I have

An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::
HttpClient::Pool::HostUnreachableError: Could not reach host Manticore::SocketException:
Connection refused (Connection refused)>, :backtrace=>["/opt///logstash/vendor
/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/
elasticsearch/http_client/pool.rb:293:in perform_request_to_url'", "/opt/****/****/logstash /vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs /elasticsearch/http_client/pool.rb:278:in block in perform_request'", "/opt///
logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/
logstash/outputs/elasticsearch/http_client/pool.rb:373:in with_connection'", " /opt/****/****/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib /logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'",
"/opt///logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in block in Pool'", "/opt/****/****/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0- java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in get'", "/opt///logstash/
ndor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/
outputs/elasticsearch/http_client.rb:378:in get_xpack_info'", "/opt/****/****/logstash/ vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/ outputs/elasticsearch/ilm.rb:57:in ilm_ready?'", "/opt///logstash/vendor/bundle
/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/
ilm.rb:28:in ilm_in_use?'", "/opt/****/****/logstash/vendor/bundle/jruby/2.5.0/gems/ logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in block in setup_after_successful_connection'"]}

Could this be a reason ?? There is no OS error

Can someone please reply back?

I see a similar error in another thread Logstash restarting by output ES. Is it existing bug?