# Logstash clustering issue **URL:** \ **Category:** Logstash\ **Created:** [October 12, 2020, 2:11pm UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777 "2020-10-12T14:11:24Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 12, 2020, 2:11pm UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/1 "2020-10-12T14:11:24Z")
Im having Elasticsearch clustering and my output.conf in logstash, looks like below ``` hosts => ["node1:9200","node2:9200","node3:9200"] ``` So through java, Im updating output.conf and starting logstash. Now in case my elasticsearch cluster nodes takes some time to come up, logstash throws an error [2020-10-12T17:52:25,998][ERROR][logstash.outputs.elasticsearch][events] Failed to install template. {:message=\>"Elasticsearch Unreachable: [[http://164.99.185.202:9200/](http://164.99.185.202:9200/)][Manticore::SocketException] Connection refused (Connection refused)", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :backtrace=\>["logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:293:in `perform_request_to_url'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform\_request'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:373:in `with_connection'", logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform\_request'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:285:in `block in Pool'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in `get'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:378:in `get_xpack_info'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:57:in `ilm\_ready?'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:28:in `ilm_in_use?'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:14:in `install\_template'", logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:130:in `install_template'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:51:in `block in setup\_after\_successful\_connection'"]} [2020-10-12T17:52:26,037][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit Is it stopped as elasticsearch nodes are down(one node is still up)?? I thought even though elasticsearch nodes are down, logstash will try to establish communication . It should not get shut down. Am I correct? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 12, 2020, 2:12pm UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/2 "2020-10-12T14:12:40Z")
Try enabling log.level debug to see if you get a more informative error message. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 12, 2020, 2:14pm UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/3 "2020-10-12T14:14:14Z")
Sure. But whats the expectation?? Even though all elasticsearch nodes are down, logstash should not stop. It should try to establish communication.Is that right??? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 12, 2020, 2:25pm UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/4 "2020-10-12T14:25:18Z")
> [@RAM\_NATHAN](#): > > `java/lib/logstash/outputs/elasticsearch/common.rb:51:in ` block in setup\_after\_successful\_connection'"]} The output thinks it has successfully connected to elasticsearch, so it is trying to do template management. It could be a race condition where if elasticsearch goes down between the test for a connection and the attempt to use it then the failure is not handled. I do not know. Generally I would expect an elasticsearch output to be OK even if all hosts are down when it is registered. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 16, 2020, 12:08pm UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/5 "2020-10-16T12:08:53Z")
This is the only Fatal error I have An unexpected error occurred! {:error=\>#\, :backtrace=\>["/opt/ **/** /logstash/vendor /bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/ elasticsearch/http\_client/pool.rb:293:in `perform_request_to_url'", "/opt/ ****/**** /logstash /vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs /elasticsearch/http_client/pool.rb:278:in `block in perform\_request'", "/opt/ **/** / logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/ logstash/outputs/elasticsearch/http\_client/pool.rb:373:in `with_connection'", " /opt/ ****/**** /logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib /logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform\_request'", "/opt/ **/** /logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0- java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:285:in `block in Pool'", "/opt/ ****/**** /logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0- java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in `get'", "/opt/ **/** /logstash/ ndor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/ outputs/elasticsearch/http\_client.rb:378:in `get_xpack_info'", "/opt/ ****/**** /logstash/ vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/ outputs/elasticsearch/ilm.rb:57:in `ilm\_ready?'", "/opt/ **/** /logstash/vendor/bundle /jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ ilm.rb:28:in `ilm_in_use?'", "/opt/ ****/**** /logstash/vendor/bundle/jruby/2.5.0/gems/ logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup\_after\_successful\_connection'"]} Could this be a reason ?? There is no OS error ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 20, 2020, 5:59am UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/6 "2020-10-20T05:59:29Z")
Can someone please reply back? ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 22, 2020, 6:40am UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/7 "2020-10-22T06:40:22Z")
I see a similar error in another thread [Logstash restarting by output ES](https://discuss.elastic.co/t/logstash-restarting-by-output-es/251509). Is it existing bug? ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 26, 2020, 8:42am UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/8 "2020-10-26T08:42:29Z")
Can you please provide an update ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [October 27, 2020, 6:17am UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/9 "2020-10-27T06:17:46Z")
After a long search, I found this article [https://github.com/elastic/logstash/issues/10722](https://github.com/elastic/logstash/issues/10722) Error in this github doesnt have HostUnreachableError: Could not reach host Manticore::SocketException: Connection refused Still should I try this solution? - "_Disable ILM in the Logstash ES output_" ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 24, 2020, 6:18am UTC](https://discuss.elastic.co/t/logstash-clustering-issue/251777/10 "2020-11-24T06:18:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.