[logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance

gerald716 · March 19, 2021, 3:21pm


[2021-03-18T15:33:29,348][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-03-18T15:33:30,951][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::Mutate", {"remove_field"=>["[metadata]"], "id"=>"6cd9545fd59682e299ddf4492604970fb6049a1785228212f812b4975ddb3241"}]=>[{"thread_id"=>162, "name"=>"[elastiflow]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>163, "name"=>"[elastiflow]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>164, "name"=>"[elastiflow]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>165, "name"=>"[elastiflow]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}]}}```

dadoonet · March 19, 2021, 4:22pm

Welcome!

What are elasticsearch logs saying?

gerald716 · March 19, 2021, 4:39pm

Dear, the data is
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.x.x.x
http.port: 9200

root@misc089des:~/elk# tail -f /var/log/elasticsearch/elasticsearch.log
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
[2021-03-19T13:22:27,285][INFO ][o.e.c.r.a.AllocationService] [1ihOYXc] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.monitoring-es-6-2021.03.19][0]] ...]).

dadoonet · March 19, 2021, 5:35pm

Could you share the full logs?
Why are you starting with a so old version?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

gerald716 · March 19, 2021, 6:07pm

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
[2021-03-19T14:16:08,472][DEBUG][o.e.a.s.TransportSearchAction] [1ihOYXc] All shards failed for phase: [query]
[2021-03-19T14:16:08,472][WARN ][r.suppressed             ] [1ihOYXc] path: /.kibana_task_manager/_doc/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:291) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:220) ~[elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:174) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:220) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:463) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1108) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1220) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1194) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:366) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:360) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:354) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1085) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.1.jar:6.7.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
[2021-03-19T14:16:08,934][INFO ][o.e.c.r.a.AllocationService] [1ihOYXc] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...]).

gerald716 · March 19, 2021, 6:09pm

It is because I found more information about elastiflow

dadoonet · April 6, 2021, 4:30pm

Could you share the full logs from the start?

AFAICS you can use Elasticsearch 7.8+ with elasticflow 4.0.x:

github.com

robcowart/elastiflow/blob/master/INSTALL.md#elastic-stack-compatibility

# ElastiFlow&trade; Installation

[![patreon](https://user-images.githubusercontent.com/10326954/52966127-c9847680-33a6-11e9-8640-10dd7abc3af0.png)](https://www.patreon.com/elastiflow) [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart) [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=ElastiFlow%E2%84%A2%20provides%20Netflow%20v5%2Fv9%2C%20sFlow%20and%20IPFIX%20data%20collection%20and%20visualization%20using%20the%20Elastic%20Stack.&url=https://github.com/robcowart/elastiflow&hashtags=elastiflow,netflow,sflow,ipfix)

ElastiFlow&trade; is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. To install and configure ElastiFlow&trade;, you must first have a working Elastic Stack environment.

> **WARNING!** - ElastiFlow 4.0.x supports Elastic Common Schema (ECS). Due to significant data model changes there is no upgrade/migration from ElastiFlow 3.x. You should either remove all 3.x indices or deploy ElastiFlow 4.0.x to a separate environment.

> **IMPORTANT!** - Always use a **RELEASE**. DO NOT use the `master` branch.

> **NOTE** - For full ElastiFlow 4.0.x functionality, including Kibana's SIEM and Logs apps, you should use X-Pack Basic or one of the commercial X-Pack tiers.

## Elastic Stack Compatibility

Refer to the following compatibility chart to choose a release of ElastiFlow&trade; that is compatible with the version of the Elastic Stack you are using.

Elastic Stack | ElastiFlow&trade; 3.x | ElastiFlow&trade; 4.x
:---:|:---:|:---:
7.8+ |  | &#10003; v4.0.x
7.5-7.7 |  | &#10003; v4.0.0-beta

This file has been truncated. show original

system · May 4, 2021, 4:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.