Heap Problem

Elastic Stack Elasticsearch
1

Hello,
we using elastic search to process analytic data, we got daily basis analytic dump from our system, we need to do some analysis on analytic data, for this we push analytic data from log files to elastic search using log stash. log stash is working fine but on elastic search when doc's indexed heap is continually grow and after some hours nodes stopped due to memory issue, so we have to restart the cluster nodes, log stash using update groovy script to increase count += some values, so if docs with a id is already exist then it will update otherwise new doc is index,

right now we are not doing any kind of query from the ES cluster,
our cluster has 4 nodes, 3 data nodes with 4 cpu and 30 GB memory,
and 1 master node

and index configuration is :

10 shards,
0 replication,
60s refresh interval

please help me to resolve the heap issue......

2

From your description it's not entirely clear to me what's going wrong. You could try and take a look at

for general advice on indexing optimisation.

Hope this helps,
Isabel

3

my problem is nodes heap is continuously grow, GC is not cleaning heap.
and i am just indexing docs

4

I am experiencing the same problem. I am indexing 400 million log lines per day on a 2 node 2.4 cluster and the heap is also slowly growing.Typically at around 95% full , I am hit by a old gc , that can take minutes to complete. This of course causing the the node to fail and potentially a cluster restart.

I have been unable to get any good advice on here so far. So I have upped my heap close to max (29 GB) and now waiting to see the results of this-