Lots of disconnected logs and then OOM

[shjdwxy]

hi,
ES version: 5.4.3
java version: 1.8.0_162
ES cluster: 3 dedicated master, 7 hot datanode, 14 cold datanode, 31GB heap for each node
9,425 indices and 23000 shards

ES cluster was running normally. But one node suddenly output lots of disconnected logs and then OOM. Before OOM， the CPU is exhausted. I am sure there is no problem with network.

logs:

https://gist.github.com/wangxiangyu/199d7171ef2f9f3d68ba27dfed94ba80

Does anyone meet this before?

[DavidTurner]

I think you already asked this, or something very like it, here:

Es node suddenly OOM with "fatal error on the network layer"

hi, ES version: 5.4.3 java version: 1.8.0_162 ES was running normally and GC is normal too. But one node suddenly OOM with "fatal error on the network layer" logs link: Does anyone meet this before? How to debug this problem ? Thank you in advance!

As before:

Es node suddenly OOM with "fatal error on the network layer"

By default Elasticsearch will write a heap dump when it encounters an OutOfMemoryError. The best thing to do is to open this heap dump (e.g. in MAT) and investigate what was using all the heap.

If you need help investigating your heap dump then you can ask more detailed questions here of course.

[shjdwxy]

ES heap is 31GB, I am wondering:

• whether the heap can be dumped successfully when oom
• even dump successfully, 31GB is too big to analyse.

[DavidTurner]

31GB is not really very large. I've never heard of that being a problem to dump, and MAT can normally open dumps of that size just fine.