# One of three pods stop but Elasticsearch thinks it's still running

**URL:** <https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158>\
**Category:** Elasticsearch\
**Created:** [November 20, 2020, 4:21pm UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158 "2020-11-20T16:21:47Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![lahpman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lahpman/32/77348_2.png) [@lahpman](https://discuss.elastic.co/u/lahpman)\
**Post date:** [November 20, 2020, 4:21pm UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158/1 "2020-11-20T16:21:48Z")

</div>

Hey y'all,

We are running Elasticsearch 7.9.3 with the help of helm-charts/elasticsearch and lately we have been having a pod go down out of the blue. When I check on the health of Elasticsearch via CURL \_cluster/health I get the following, which infers all nodes are ok :

```auto
      "cluster_name" : "elasticsearch",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 3,
      "number_of_data_nodes" : 3,
      "active_primary_shards" : 43,
      "active_shards" : 86,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0

```

I've checked the pod logs to see what happens before it dies, but this is all I get:

```
    [gc][147189] overhead, spent [2.9s] collecting in the last [5.2s]
    [gc][young][147189][871] duration [2.9s], collections [1]/[5.2s], total [2.9s]/[56.1s], memory 
    [762.6mb]->[150.7mb]/[1gb], all_pools {[young] [612mb]->[1mb]/[0b]}{[old] [150.1mb]-> 
    [150.1mb]/[1gb]}{[survivor] [534.3kb]->[614.8kb]/[0b]}
    [gc][146383] overhead, spent [669ms] collecting in the last [1.1s]
    [gc][137869] overhead, spent [261ms] collecting in the last [1s]
    [gc][126549] overhead, spent [287ms] collecting in the last [1s]

```

Any ideas? I did some searching and saw that the above messages were related to heap management, but I'm not so sure.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [November 22, 2020, 1:21am UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158/2 "2020-11-22T01:21:41Z")

</div>

2.9 of 5.2 seconds and .669 of 1.1 is a lot if time the node will do nothing. It does "stop the world" for that time. Is it possible that it's just pausing and recovering or do you see a full node restart in the log? How much heap do you have allocated?

---

<div class="post-metadata">

**Author:** ![lahpman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lahpman/32/77348_2.png) [@lahpman](https://discuss.elastic.co/u/lahpman)\
**Post date:** [November 23, 2020, 1:05pm UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158/3 "2020-11-23T13:05:02Z")

</div>

Hey thanks for the response,

I don't think it attempts a restart, it just sort of stops, and when checking pod status it says it's running but it is no longer "Ready". When I try to exec into the pod it's unresponsive, and the exec command fails.

As for the heap, we are using the helm-chart default, which is 1 gig.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [November 23, 2020, 4:40pm UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158/4 "2020-11-23T16:40:44Z")

</div>

How old is this cluster? Did it work OK for awhile, but more data made it unhealthy? Limited to 1G heap, it may be time to add nodes.

---

<div class="post-metadata">

**Author:** ![lahpman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lahpman/32/77348_2.png) [@lahpman](https://discuss.elastic.co/u/lahpman)\
**Post date:** [November 23, 2020, 5:44pm UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158/5 "2020-11-23T17:44:29Z")

</div>

We started off on Elasticsearch 6.8.X last summer (2019) and upgraded to 7 in January. We had no real issue until our upgrade to 7.9 (from 7.4.X I believe). From 7.9.0 we thought maybe it was the memory leak reported here [https://github.com/elastic/elasticsearch/issues/61512](https://github.com/elastic/elasticsearch/issues/61512) . So we did an upgrade to 7.9.3 and the issue still seems to occur.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2020, 5:44pm UTC](https://discuss.elastic.co/t/one-of-three-pods-stop-but-elasticsearch-thinks-its-still-running/256158/6 "2020-12-21T17:44:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.