Metricbeat is unable to send data to elastic search(011Attempting to reconnect to backoff(elasticsearch(http://x.x.x.x:9200)) with 441 reconnect attempt(s))

I have recently updated ES, Kibana, Logstash, All the services are running but the data is not getting collected. and UI is also loading. Please help me to resolve this.

Kibana Log:
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["info","savedobjects-service"],"pid":18273,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["info","savedobjects-service"],"pid":18273,"message":"Creating index .kibana_task_manager_3."}
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["info","savedobjects-service"],"pid":18273,"message":"Creating index .kibana_3."}
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["warning","savedobjects-service"],"pid":18273,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_3/37_qO60MRAOHhCGjCgz7RA] already exists, with { index_uuid="37_qO60MRAOHhCGjCgz7RA" & index=".kibana_task_manager_3" }"}
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["warning","savedobjects-service"],"pid":18273,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_3 and restarting Kibana."}
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["warning","savedobjects-service"],"pid":18273,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_3/ej5RzANpSr-L86WUfe6PFA] already exists, with { index_uuid="ej5RzANpSr-L86WUfe6PFA" & index=".kibana_3" }"}
{"type":"log","@timestamp":"2020-09-27T14:19:52Z","tags":["warning","savedobjects-service"],"pid":18273,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting Kibana."}
{"type":"log","@timestamp":"2020-09-27T18:25:40Z","tags":["info","cli","config"],"pid":18273,"message":"Reloading logging configuration due to SIGHUP."}
{"type":"log","@timestamp":"2020-09-27T18:25:40Z","tags":["info","cli","config"],"pid":18273,"message":"Reloaded logging configuration due to SIGHUP."}
{"type":"log","@timestamp":"2020-09-27T18:25:40Z","tags":["info","plugins-system"],"pid":18273,"message":"Stopping all plugins."}

ES Log:

[2020-09-27T13:37:17,968][WARN ][r.suppressed ] [node-1] path: /.kibana_task_manager/_count, params: {index=.kibana_task_manager}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:223) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:288) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]

You haven't provided anything that shows Metricbeat isn't sending data to Elasticsearch, so it's hard to say what the issue is.

The warning you show shouldn't stop Metricbeat.

Here is the log.

Sep 27 03:47:29 li2111-133 metricbeat: 2020-09-27T03:47:29.476Z#011ERROR#011pipeline/output.go:100#011Failed to connect to backoff(elasticsearch(http://x.x.x.x:9200)): Get http://x.x.x.x:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 27 03:47:29 li2111-133 metricbeat: 2020-09-27T03:47:29.476Z#011INFO#011pipeline/output.go:93#011Attempting to reconnect to backoff(elasticsearch(http://x.x.x.x:9200)) with 441 reconnect attempt(s)
Sep 27 03:47:29 li2111-133 metricbeat: 2020-09-27T03:47:29.476Z#011INFO#011[publisher]#011pipeline/retry.go:189#011retryer: send unwait-signal to consumer
Sep 27 03:47:29 li2111-133 metricbeat: 2020-09-27T03:47:29.476Z#011INFO#011[publisher]#011pipeline/retry.go:191#011 done
Sep 27 03:47:29 li2111-133 metricbeat: 2020-09-27T03:47:29.476Z#011INFO#011[publisher]#011pipeline/retry.go:166#011retryer: send wait signal to consumer
Sep 27 03:47:29 li2111-133 metricbeat: 2020-09-27T03:47:29.476Z#011INFO#011[publisher]#011pipeline/retry.go:168#011 done

What is the output from the _cat/health API on Elasticsearch?

1601267052 04:24:12 elasticsearch red 1 1 308 308 0 3 64 1 - 82.1%

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1601267635 04:33:55 elasticsearch yellow 1 1 366 366 0 0 9 0 - 97.6%

Ok that's good, it means that the cluster is recovering.
Is Metricbeat sending now?

No. Not yet. but

Port 9200 itself is not connecting from any other machines on which the metricbeat is installed(no firewall restrictions).

Do you mean you cannot connect to port 9200 on Elasticsearch?

Yes.,from remote machines(metricbeat)

Curl output:
curl: (7) Failed connect to 169.38.68.40:9200; Connection timed out

Can you run a curl to the Elasticsearch host from one of the machines?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.