# Elastic data nodes are randomly getting unresponsive

**URL:** <https://discuss.elastic.co/t/elastic-data-nodes-are-randomly-getting-unresponsive/291793>\
**Category:** Elasticsearch\
**Created:** [December 14, 2021, 12:21pm UTC](https://discuss.elastic.co/t/elastic-data-nodes-are-randomly-getting-unresponsive/291793 "2021-12-14T12:21:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fschoell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fschoell/32/98899_2.png) [@fschoell](https://discuss.elastic.co/u/fschoell)\
**Post date:** [December 14, 2021, 12:21pm UTC](https://discuss.elastic.co/t/elastic-data-nodes-are-randomly-getting-unresponsive/291793/1 "2021-12-14T12:21:21Z")

</div>

Hello, we run an Elasticsearch cluster with 12 data nodes of which groups of 4 share a larger dom server and are run in linux containers (dom servers are currently 48 core CPUs with 1.5 TB of memory and large SSD arrays). We are increasingly having issues that some of those data nodes are getting unresponsive, which means that API requests collecting node stats are timing out (such as `GET _cluster/stats` or `GET _nodes/_all`). This often (but not always) will impact data injection and our filebeat instances might run behind indexing our nginx logs. The `GET _cat/health?v` keeps reporting the cluster as healthy. Impacted nodes also have a significantly higher CPU usage as those not impacted, but CPU will drop after the Elasticsearch service has been restarted.

Looking at the logs of the affected data nodes it seems to be an issue with the node stats collector:

```auto
[2021-12-14T11:36:18,219][ERROR][o.e.x.m.c.n.NodeStatsCollector] [esh1-data13c] collector [node_stats] timed out when collecting data: node [oTyaaQiuQk-7osC9yirMwQ] did not respond within [30s]

```

I've created a [Gist](https://gist.github.com/fschoell/9ec4914893f13b2950cd48b0188a24cc) with different results from the Elastic API, including:

[GET \_cluster/stats](https://gist.github.com/fschoell/9ec4914893f13b2950cd48b0188a24cc#file-elastic_cluster_stats)

[GET \_nodes/hot\_threads?threads=9999](https://gist.github.com/fschoell/9ec4914893f13b2950cd48b0188a24cc#file-elastic_hot_threads) (on that request the two data nodes impacted are [esh1-data13b](https://gist.github.com/fschoell/9ec4914893f13b2950cd48b0188a24cc#file-elastic_hot_threads-L660) and [esh1-data13c](https://gist.github.com/fschoell/9ec4914893f13b2950cd48b0188a24cc#file-elastic_hot_threads-L214))

Any idea where this issue might come from?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2022, 12:21pm UTC](https://discuss.elastic.co/t/elastic-data-nodes-are-randomly-getting-unresponsive/291793/2 "2022-01-11T12:21:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.