# \[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance

**URL:** <https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [March 19, 2021, 3:21pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826 "2021-03-19T15:21:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gerald716](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gerald716/32/85816_2.png) [@gerald716](https://discuss.elastic.co/u/gerald716)\
**Post date:** [March 19, 2021, 3:21pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/1 "2021-03-19T15:21:45Z")

</div>

````auto

[2021-03-18T15:33:29,348][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-03-18T15:33:30,951][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::Mutate", {"remove_field"=>["[metadata]"], "id"=>"6cd9545fd59682e299ddf4492604970fb6049a1785228212f812b4975ddb3241"}]=>[{"thread_id"=>162, "name"=>"[elastiflow]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>163, "name"=>"[elastiflow]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>164, "name"=>"[elastiflow]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>165, "name"=>"[elastiflow]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}]}}```
````

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 19, 2021, 4:22pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/2 "2021-03-19T16:22:25Z")

</div>

Welcome!

What are elasticsearch logs saying?

---

<div class="post-metadata">

**Author:** ![gerald716](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gerald716/32/85816_2.png) [@gerald716](https://discuss.elastic.co/u/gerald716)\
**Post date:** [March 19, 2021, 4:39pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/3 "2021-03-19T16:39:07Z")

</div>

Dear, the data is  
path.data: /var/lib/elasticsearch  
path.logs: /var/log/elasticsearch  
network.host: 10.x.x.x  
http.port: 9200

root@misc089des:~/elk# tail -f /var/log/elasticsearch/elasticsearch.log  
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.7.1.jar:6.7.1]  
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.7.1.jar:6.7.1]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]  
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.1.jar:6.7.1]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.1.jar:6.7.1]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_282]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_282]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_282]  
[2021-03-19T13:22:27,285][INFO][o.e.c.r.a.AllocationService] [1ihOYXc] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.monitoring-es-6-2021.03.19][0]] ...]).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 19, 2021, 5:35pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/4 "2021-03-19T17:35:28Z")

</div>

Could you share the full logs?  
Why are you starting with a so old version?

Please format your code, logs or configuration files using `</>` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and not the citation button. It will make your post more readable.

Or use markdown style like:

````
```
CODE
```

````

This is the icon to use if you are not using markdown format:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e6e239431ec2d71cbf1beef741f2e93e7cc762c.jpg)

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.  
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.  
Please update your post.

---

<div class="post-metadata">

**Author:** ![gerald716](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gerald716/32/85816_2.png) [@gerald716](https://discuss.elastic.co/u/gerald716)\
**Post date:** [March 19, 2021, 6:07pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/5 "2021-03-19T18:07:54Z")

</div>

```
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
[2021-03-19T14:16:08,472][DEBUG][o.e.a.s.TransportSearchAction] [1ihOYXc] All shards failed for phase: [query]
[2021-03-19T14:16:08,472][WARN][r.suppressed] [1ihOYXc] path: /.kibana_task_manager/_doc/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:291) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:220) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:174) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:220) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:463) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1108) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1220) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1194) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:366) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:360) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:354) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1085) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
[2021-03-19T14:16:08,934][INFO][o.e.c.r.a.AllocationService] [1ihOYXc] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...]).
```

---

<div class="post-metadata">

**Author:** ![gerald716](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gerald716/32/85816_2.png) [@gerald716](https://discuss.elastic.co/u/gerald716)\
**Post date:** [March 19, 2021, 6:09pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/6 "2021-03-19T18:09:00Z")

</div>

It is because I found more information about elastiflow

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 6, 2021, 4:30pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/7 "2021-04-06T16:30:31Z")

</div>

Could you share the **full** logs from the start?

> [@gerald716](#):
>
> It is because I found more information about elastiflow

AFAICS you can use Elasticsearch 7.8+ with elasticflow 4.0.x:

> <https://github.com/robcowart/elastiflow/blob/master/INSTALL.md#elastic-stack-compatibility>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 4, 2021, 4:30pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/267826/8 "2021-05-04T16:30:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.