# ES heap space is always at 90% for all the 11 nodes having 20GB JVM space allocated (out of 28 GB) with 4 core CPU

**URL:** <https://discuss.elastic.co/t/es-heap-space-is-always-at-90-for-all-the-11-nodes-having-20gb-jvm-space-allocated-out-of-28-gb-with-4-core-cpu/188488>\
**Category:** Elasticsearch\
**Created:** [July 2, 2019, 11:07am UTC](https://discuss.elastic.co/t/es-heap-space-is-always-at-90-for-all-the-11-nodes-having-20gb-jvm-space-allocated-out-of-28-gb-with-4-core-cpu/188488 "2019-07-02T11:07:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manoj\_Chhetri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manoj_chhetri/32/49262_2.png) [@Manoj\_Chhetri](https://discuss.elastic.co/u/Manoj_Chhetri)\
**Post date:** [July 2, 2019, 11:07am UTC](https://discuss.elastic.co/t/es-heap-space-is-always-at-90-for-all-the-11-nodes-having-20gb-jvm-space-allocated-out-of-28-gb-with-4-core-cpu/188488/1 "2019-07-02T11:07:05Z")

</div>

I have a 11 node ES cluster. Each node has

> 1. 4 core cpu
> 2. Total memory 28GB (To ES i allocated 20 GB for each node)
> 3. 1 OS disk 2TB , 2 attached data disk of 2TB =\> total 6 TB each ES node

Total cluster disk space 6 \* 11 -\> 66 TB Currently i have almost 25 TB data with

```auto
1. 224 Indices
2. 1059 shards
3. I have nested document so for each record multiple entries is created , currently total number of docs inserted are 596915451283 docks (596 bn records)

```

For each node settings applied :-

```auto
**/etc/elasticsearch/elasticsearch.yml**
discovery.zen.ping_timeout: 15s
discovery.zen.publish_timeout: 60s
discovery.zen.fd.ping_interval: 5s
discovery.zen.fd.ping_retries: "3"
discovery.zen.fd.ping_timeout: 15s
discovery.zen.join_timeout: 1m
discovery.zen.minimum_master_nodes: "1"

bootstrap.memory_lock: true
indices.fielddata.cache.size: 5%
indices.queries.cache.size: 2%
indices.breaker.fielddata.limit: 50%

**/etc/elasticsearch/jvm.options**
all default config only added to demand 20gb mem from OS for ES
-Xms20g
-Xmx20g

**/etc/default/elasticsearch**
MAX_LOCKED_MEMORY=unlimited
MAX_OPEN_FILES=65536

**/usr/lib/systemd/system/elasticsearch.service**
LimitMEMLOCK=infinity under [Service] tag

**/etc/security/limits.conf**
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

```

`ulimit -n 65536`

So i think , i have applied all the ES optimisation config i found over the blog and document by ES

Problem I am facing :- Even if there is no query is running at all ,But yes there is write operation happening from Kafka Elasticsearch connector say every 30 sec interval. ES is burning (17-18 GB) 90% of heap out of 20GB allocated.

Tell me like

```auto
1. I am missing any config which need to be tuned.
2. If i am not missing , then do we have only option to upgrade ES cluster (updating the current node or adding the new node)
3. If its JVM thing need to tuned so how i can do that.

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2019, 11:07am UTC](https://discuss.elastic.co/t/es-heap-space-is-always-at-90-for-all-the-11-nodes-having-20gb-jvm-space-allocated-out-of-28-gb-with-4-core-cpu/188488/2 "2019-07-30T11:07:09Z")

</div>

Closed on July 30, 2019, 11:07am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.