OOM for ES 0.19.7

Elastic Stack Elasticsearch
1

hi,all,

  • *I have 4 es node with 32 shard 1 replica, each with 8G memory(no
    mlock). my elasticsearch version is elasticsearch-0.19.7(recently upgraded
    from 0.18.5).
    I start each node with command:
    nohup bin/elasticsearch -Xms8g -Xmx8g -Des.max-open-files=true
    -Dbootstrap.mlockall=true -p es.pid

as umlit -l prints 8388608, that is 8g, the above command prints log Unknown
mlockall error 0 and as a result actually i started es with no mlock.

last night, i received bunch of alarm smses of this service and i logged in
and found that 3 of the 4 nodes are out of memory and gc logs is crazily
rolling.

and current the index size is:
size: 35.8gb (71.7gb)
docs: 16250075 (17764421)

good news different from 0.18.5, the cpu useage drops off to around 100%
not 200+% any more.

as I have no root access to the production machine, I now restart each node
with command
nohup bin/elasticsearch -Xms7g -Xmx7g -Des.max-open-files=true
-Dbootstrap.mlockall=true -p es.pid

does anyone has similar experience and know how to fix this problem
for periodical OOM hangs.

thanks ~