# Error while performing resurretion after restarting elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [October 26, 2016, 9:47am UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980 "2016-10-26T09:47:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![nick.e](https://avatars.discourse-cdn.com/v4/letter/n/8dc957/32.png) [@nick.e](https://discuss.elastic.co/u/nick.e)\ **Post date:** [October 26, 2016, 9:47am UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/1 "2016-10-26T09:47:28Z")
Hello together, I installed a single-node-cluster with the 5.0.0rc1 version of the ELK-Stack for some testing. When restarting the elasticsearch-service logstash tries to reconnect: > [ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@127.0.0.1:9200][Manticore::SocketException] Broken pipe", :class=\>"LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>2} > [2016-10-26T11:32:32,134][WARN][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=\>#\} But then, when elasticsearch itself startet properly and i can access it by kibana or http, logstash throws another error: > [2016-10-26T11:33:15,097][WARN][logstash.outputs.elasticsearch] Error while performing resurrection {:error\_message=\>"Got response code '403' contact Elasticsearch at URL 'http://~hidden~:~hidden~@127.0.0.1:9200/'", :class=\>"LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError", :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http\_client/manticore\_adapter.rb:48:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:233:in `perform\_request\_to\_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:201:in `resurrect_dead!'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:196:in `resurrect_dead!'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:189:in `start\_resurrectionist'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:117:in `until_stopped'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.1.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188:in `start\_resurrectionist'"]} > [2016-10-26T11:33:20,100][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=\>#\, :healthcheck\_path=\>"/"} This error is occuring all the time and logstash won't reconnect. When trying to restart logstash itself the service won't stop and has to be force-killed. (Well, I think this could be proper behaviour to not lose any data) After starting it again, everything works fine and the elasticsearch-outputs connect correctly. Elasticsearch is protected by security. The logstash-elasticsearch-outputs look like this: > elasticsearch { > index =\> "test-index-%{+YYYY.MM.dd}" > user =\> "logstash" > password =\> "thepassword" > } Anyone an idea what to do? ---
**Author:** ![nick.e](https://avatars.discourse-cdn.com/v4/letter/n/8dc957/32.png) [@nick.e](https://discuss.elastic.co/u/nick.e)\ **Post date:** [October 27, 2016, 3:21pm UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/2 "2016-10-27T15:21:24Z")
Additional information: Upgraded the stack to the release of 5.0.0 today, still encountering the same problem. The Logstash output won't reconnect. ---
**Author:** ![Tribbles](https://avatars.discourse-cdn.com/v4/letter/t/9fc348/32.png) [@Tribbles](https://discuss.elastic.co/u/Tribbles)\ **Post date:** [November 16, 2016, 7:30am UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/3 "2016-11-16T07:30:15Z")
You are probably using shield/x-pack for authentication. You need to make sure that your Logstash user has sufficient privileges. Try this with curl, i.e. curl -u logstash:thepassword [http://127.0.0.1:9200](http://127.0.0.1:9200) (as per the log output with the hidden pw entry). Most likely you won't receive an ES status JSON, but an error. To solve this, check your roles,yaml file for the logstash user and make sure an entry like: cluster: ['monitor'] exists. Then retry access with above curl-command. Best wishes, Thorsten ---
**Author:** ![nick.e](https://avatars.discourse-cdn.com/v4/letter/n/8dc957/32.png) [@nick.e](https://discuss.elastic.co/u/nick.e)\ **Post date:** [November 16, 2016, 9:11am UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/4 "2016-11-16T09:11:57Z")
Hey Thorsten, thanks. that solved my problem. I had to add the "Monitor" cluster-privilege to my logstash-role. ---
**Author:** ![simo](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@simo](https://discuss.elastic.co/u/simo)\ **Post date:** [April 12, 2017, 10:02am UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/5 "2017-04-12T10:02:09Z")
hey Nick, i have the same problem how i can use this solution ?? ---
**Author:** ![nick.e](https://avatars.discourse-cdn.com/v4/letter/n/8dc957/32.png) [@nick.e](https://discuss.elastic.co/u/nick.e)\ **Post date:** [April 12, 2017, 12:17pm UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/6 "2017-04-12T12:17:36Z")
> [@simo](#): > > hey Nick,i have the same problem how i can use this solution ?? This solved the Problem: > [@Tribbles](#): > > To solve this, check your roles,yaml file for the logstash user and make sure an entry like: > > cluster: ['monitor'] > > exists Give your user which you use in logstash to send data into elasticsearch the "Monitor" Cluster Privilege. > **[Defining Roles | X-Pack for the Elastic Stack \[6.2\] | Elastic](https://www.elastic.co/guide/en/x-pack/current/defining-roles.html)** ---
**Author:** ![simo](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@simo](https://discuss.elastic.co/u/simo)\ **Post date:** [April 12, 2017, 2:47pm UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/7 "2017-04-12T14:47:58Z")
thanks nick i found the incorrect password of elasticsearch thanks for your response ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:27am UTC](https://discuss.elastic.co/t/error-while-performing-resurretion-after-restarting-elasticsearch/63980/8 "2017-07-06T04:27:08Z")