# ElasticSearch nodes not responding anymore - please help!

**URL:** https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972
**Category:** Elasticsearch
**Created:** [September 7, 2016, 10:41am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972 "2016-09-07T10:41:08Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![renego](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 7, 2016, 10:41am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/1 "2016-09-07T10:41:08Z")

</div>

Hi, I have the problem, that until some days of running the nodes of ES doesn't respond anymore until I restart them. I have installed for this reason Marvel to take a look into the system.

I am attaching the Marvel stats. I hope you can help me!

[[Album] Imgur: The magic of the Internet ![](https://i.imgur.com/od89uoU.jpg?fb "Imgur: The magic of the Internet") ](https://imgur.com/a/KAAcv)

On the image you can see, that when the JVM Heap reaches \> 75%, then the load is going high and the nodes are dead. They can not respond anymore.

I had in my settings for each node an heap of 12g - ES\_HEAP\_SIZE=12g . Now I have reduce it to 8g to see what will happen.

Does anyone has an idea where the problem can be? Do you need some other info from my settings?

Thanks  
Nik

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 7, 2016, 10:45am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/2 "2016-09-07T10:45:17Z")

</div>

Is there anything interesting in the logs?

---

<div class="post-metadata">

### Author: ![renego](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 7, 2016, 10:48am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/3 "2016-09-07T10:48:15Z")

</div>

I have many logs from this:

`[2016-09-06 23:21:32,056][WARN][transport] [xxx-master-1] Received response for a request that has timed out, sent [19090ms] ago, timed out [4090ms] ago, actio`

---

<div class="post-metadata">

### Author: ![renego](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 7, 2016, 2:44pm UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/4 "2016-09-07T14:44:33Z")

</div>

Are this graphs of the 2 nodes for the JVM Heap for the last 4 hours looking good?

[[Album] Imgur: The magic of the Internet ![](https://i.imgur.com/A2MLFZP.jpg?fb "Imgur: The magic of the Internet") ](https://imgur.com/a/7YM2I)

Thanks  
Nik

---

<div class="post-metadata">

### Author: ![kstaken](https://avatars.discourse-cdn.com/v4/letter/k/e68b1a/32.png) [@kstaken](https://discuss.elastic.co/u/kstaken)
#### Post date: [September 8, 2016, 9:31pm UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/5 "2016-09-08T21:31:42Z")

</div>

That looks like a fairly normal garbage collection cycle but you may want to ask the question of what aspect of your workload is driving up the heap utilization so quickly.

---

<div class="post-metadata">

### Author: ![renego](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 9, 2016, 10:13am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/6 "2016-09-09T10:13:58Z")

</div>

Hi kstaken, at the moment where you see the increase of the heap I am rebuilding the index. I create a new index, fill the data and delete the old one. This happen every day.

I am wondering, why the heap is not empty by the delete process. Should I somehow empty the heap in the moment where I am deleting the old index?

---

<div class="post-metadata">

### Author: ![renego](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 9, 2016, 12:15pm UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/7 "2016-09-09T12:15:14Z")

</div>

Here are my data from HQ. Do someone needs more info?

[[Album] Imgur: The magic of the Internet ![](https://i.imgur.com/MeUQiuX.jpg?fb "Imgur: The magic of the Internet") ](https://imgur.com/a/hClrN)

Please can someone help me? My cluster is not responding every 3 days and this is causing big problems on my sites.

Thanks  
Nik

---

<div class="post-metadata">

### Author: ![kstaken](https://avatars.discourse-cdn.com/v4/letter/k/e68b1a/32.png) [@kstaken](https://discuss.elastic.co/u/kstaken)
#### Post date: [September 9, 2016, 3:51pm UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/8 "2016-09-09T15:51:28Z")

</div>

How much physical memory is on the node relative to the heap size? How many nodes? Master and data nodes separate or combined?

Maybe your reindexing process is simply running too aggressively and needs to be throttled back some.

---

<div class="post-metadata">

### Author: ![renego](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 8:53am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/9 "2016-09-11T08:53:45Z")

</div>

Hi, here are my server and ES settings:

> [@Please suggest performance settings for my ElasticSearch cluster](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232):
>
> Hi, I have troubles with my ElasticSearch cluster and don't know what to do. Here my settings: Server: 64GB of RAM 8 cores 500GB SSD Data: 10 different index each 1M rows 10M of rows 11GB of data Actual ES cluster 3 nodes on the same server 1 master node and 2 slave nodes 8g Heap size for each node bootstrap.mlockall: true Search requests: Search Rate: 150 /s Search Latency: 1ms I have a lot of aggregations and filter for each search request Problem: When the JVM heap is comin…

Cheers  
Nik

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:21pm UTC](https://discuss.elastic.co/t/elasticsearch-nodes-not-responding-anymore-please-help/59972/10 "2017-07-05T22:21:20Z")

</div>


