Lots of disconnected logs and then OOM

hi,
ES version: 5.4.3
java version: 1.8.0_162
ES cluster: 3 dedicated master, 7 hot datanode, 14 cold datanode, 31GB heap for each node
9,425 indices and 23000 shards

ES cluster was running normally. But one node suddenly output lots of disconnected logs and then OOM. Before OOM, the CPU is exhausted. I am sure there is no problem with network.

logs:

Does anyone meet this before?

I think you already asked this, or something very like it, here:

As before:

If you need help investigating your heap dump then you can ask more detailed questions here of course.

ES heap is 31GB, I am wondering:

  • whether the heap can be dumped successfully when oom
  • even dump successfully, 31GB is too big to analyse.

:joy:

31GB is not really very large. I've never heard of that being a problem to dump, and MAT can normally open dumps of that size just fine.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.