# Lots of disconnected logs and then OOM

**URL:** <https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758>\
**Category:** Elasticsearch\
**Created:** [April 21, 2019, 2:39pm UTC](https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758 "2019-04-21T14:39:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shjdwxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shjdwxy/32/43102_2.png) [@shjdwxy](https://discuss.elastic.co/u/shjdwxy)\
**Post date:** [April 21, 2019, 2:39pm UTC](https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758/1 "2019-04-21T14:39:09Z")

</div>

hi,  
ES version: 5.4.3  
java version: 1.8.0\_162  
ES cluster: 3 dedicated master, 7 hot datanode, 14 cold datanode, 31GB heap for each node  
9,425 indices and 23000 shards

ES cluster was running normally. But one node suddenly output lots of disconnected logs and then OOM. Before OOM， the CPU is exhausted. I am sure there is no problem with network.

logs:

> <https://gist.github.com/wangxiangyu/199d7171ef2f9f3d68ba27dfed94ba80>

Does anyone meet this before?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 21, 2019, 3:26pm UTC](https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758/2 "2019-04-21T15:26:12Z")

</div>

I think you already asked this, or something very like it, here:

> [@Es node suddenly OOM with "fatal error on the network layer"](https://discuss.elastic.co/t/es-node-suddenly-oom-with-fatal-error-on-the-network-layer/177746/1):
>
> hi, ES version: 5.4.3 java version: 1.8.0\_162 ES was running normally and GC is normal too. But one node suddenly OOM with "fatal error on the network layer" logs link: Does anyone meet this before? How to debug this problem ? Thank you in advance!

As before:

> [@Es node suddenly OOM with "fatal error on the network layer"](https://discuss.elastic.co/t/es-node-suddenly-oom-with-fatal-error-on-the-network-layer/177746/2):
>
> By default Elasticsearch will write a heap dump when it encounters an OutOfMemoryError. The best thing to do is to open this heap dump (e.g. in [MAT](https://www.eclipse.org/mat/)) and investigate what was using all the heap.

If you need help investigating your heap dump then you can ask more detailed questions here of course.

---

<div class="post-metadata">

**Author:** ![shjdwxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shjdwxy/32/43102_2.png) [@shjdwxy](https://discuss.elastic.co/u/shjdwxy)\
**Post date:** [April 22, 2019, 8:40am UTC](https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758/3 "2019-04-22T08:40:58Z")

</div>

ES heap is 31GB, I am wondering:

- whether the heap can be dumped successfully when oom
- even dump successfully, 31GB is too big to analyse.

😂

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 22, 2019, 9:17am UTC](https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758/4 "2019-04-22T09:17:06Z")

</div>

31GB is not really very large. I've never heard of that being a problem to dump, and MAT can normally open dumps of that size just fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 20, 2019, 9:17am UTC](https://discuss.elastic.co/t/lots-of-disconnected-logs-and-then-oom/177758/5 "2019-05-20T09:17:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.