# How to optimise heap usage on elasticsearch nodes?

**URL:** <https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627>\
**Category:** Elasticsearch\
**Created:** [October 10, 2020, 4:30am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627 "2020-10-10T04:30:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nitish\_Goyal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitish_goyal/32/54988_2.png) [@Nitish\_Goyal](https://discuss.elastic.co/u/Nitish_Goyal)\
**Post date:** [October 10, 2020, 4:30am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/1 "2020-10-10T04:30:45Z")

</div>

We use elasticsearch for analytics and ingest around 3 billions events a day. For this, we create daily indices and each index has a ttl of 30 days. We flush out older indices through a cron job

Cluster Size : 200 nodes

Each node has below configuration:  
Ram : 256 GB  
Heap : 30 GB  
Cores : 40  
180-200 shards per node

Few of our daily indices have 80 shards each and each shard contains 20GB to 30 GB of data

We regularly see our heap usage remains at ~75% usage

1. If I reduce shards per index from 80 to 60, which would increase size of each shard from 30 to 40 GB, will it reduce pressure on heap?  
I am assuming each node will have fewer shards and hence fewer inverted indices on each node

2. As per this article, [https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage](https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage), will moving to 7.7 help reduce pressure on heap?

Kindly help on the above, our nodes keep going down because of pressure on heap

Regards,  
Nitish Goyal

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 10, 2020, 4:49am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/2 "2020-10-10T04:49:35Z")

</div>

Which version are you currently on? Going to the latest version should reduce heap usage but doing so on a 200 node cluster may take a while, especially if you are on an older version.

If you have fast disks you can try to forcemerge indices that are no longer written to down to a single segment as this has the potential to save quite a bit of heap. This may help you without having to migrate, but exactly how much it helps depends on your data.

---

<div class="post-metadata">

**Author:** ![Nitish\_Goyal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitish_goyal/32/54988_2.png) [@Nitish\_Goyal](https://discuss.elastic.co/u/Nitish_Goyal)\
**Post date:** [October 10, 2020, 5:13am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/3 "2020-10-10T05:13:17Z")

</div>

We are currently using 6.8 and 3 months back did a rolling upgrade from 6.3.0. Each node has 6T disk attached and we have around 600 TB data in the cluster

We already have a cron job in place which force merges segments for indices not actively written

Shall we look for ?

1. Reducing shards per index? Read in a blog that each shard should have less than 50GB of data
2. If not option 1, shall we go for shrinking indices?

Can you suggest any other recommendations?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 10, 2020, 8:21am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/4 "2020-10-10T08:21:20Z")

</div>

> [@Nitish\_Goyal](#):
>
> We already have a cron job in place which force merges segments for indices not actively written

That is good to hear.

> [@Nitish\_Goyal](#):
>
> Reducing shards per index? Read in a blog that each shard should have less than 50GB of data

I do not think this will help much. You have a reasonable shard sizer and a reasonable number of shards per node.

> [@Nitish\_Goyal](#):
>
> If not option 1, shall we go for shrinking indices?

I do not think this will help either as your shards are already reasonably large and you are forcemerging them down to a single segment.

> [@Nitish\_Goyal](#):
>
> Can you suggest any other recommendations?

As you have large hosts I was initially going to suggest having more than one node per host, but given that you already have 200 nodes and disks are reasonably full, such migration would be difficult and increase the number of nodes in the cluster beyond what I would generally recommend.

Another way would be to adopt using GIGC which has support for larger heaps. It requires a recent Java version and possibly also a newer version of Elasticsearch. A larger heap would however mean you no longer benefit from compressed pointers and have a smaller file system page cache which reduces the potential gain.

It would be good to get a better idea of the cluster and see if there is anything unusual. Could you please provide the fill output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/cluster-stats.html)? Not sure there is any big gains to make, so upgrading to the latest version might be the best way forward.

---

<div class="post-metadata">

**Author:** ![Nitish\_Goyal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitish_goyal/32/54988_2.png) [@Nitish\_Goyal](https://discuss.elastic.co/u/Nitish_Goyal)\
**Post date:** [October 13, 2020, 6:16am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/5 "2020-10-13T06:16:27Z")

</div>

Cluster Stats API

```auto
{
_nodes: {
total: 211,
successful: 211,
failed: 0
},
cluster_name: "prd-xx",
cluster_uuid: "xxx",
timestamp: 1602565799094,
status: "green",
indices: {
count: 4588,
shards: {
total: 35848,
primaries: 17924,
replication: 1,
index: {
shards: {
min: 2,
max: 160,
avg: 7.813426329555361
},
primaries: {
min: 1,
max: 80,
avg: 3.9067131647776807
},
replication: {
min: 1,
max: 1,
avg: 1
}
}
},
docs: {
count: 194556935778,
deleted: 163123439
},
store: {
size_in_bytes: 503410814095348
},
fielddata: {
memory_size_in_bytes: 289599575936,
evictions: 767911
},
query_cache: {
memory_size_in_bytes: 516833416064,
total_count: 299279498997,
hit_count: 10647121478,
miss_count: 288632377519,
cache_size: 14126195,
cache_count: 103558469,
evictions: 89432274
},
completion: {
size_in_bytes: 0
},
segments: {
count: 860459,
memory_in_bytes: 1521779458155,
terms_memory_in_bytes: 1429778014307,
stored_fields_memory_in_bytes: 49775618856,
term_vectors_memory_in_bytes: 0,
norms_memory_in_bytes: 8479335168,
points_memory_in_bytes: 28213633220,
doc_values_memory_in_bytes: 5532856604,
index_writer_memory_in_bytes: 60621898486,
version_map_memory_in_bytes: 780473520,
fixed_bit_set_memory_in_bytes: 2400,
max_unsafe_auto_id_timestamp: 1573720765787,
file_sizes: { }
}
},
nodes: {
count: {
total: 211,
data: 198,
coordinating_only: 0,
master: 3,
ingest: 211
},
versions: [
"6.8.10"
],
os: {
available_processors: 3340,
allocated_processors: 3340,
names: [
{
name: "Linux",
count: 211
}
],
pretty_names: [
{
pretty_name: "Ubuntu 16.04.5 LTS",
count: 211
}
],
mem: {
total_in_bytes: 18890354393088,
free_in_bytes: 313250426880,
used_in_bytes: 18577103966208,
free_percent: 2,
used_percent: 98
}
},
process: {
cpu: {
percent: 4009
},
open_file_descriptors: {
min: 5047,
max: 6302,
avg: 6030
}
},
jvm: {
max_uptime_in_millis: 9547783916,
versions: [
{
version: "1.8.0_252",
vm_name: "OpenJDK 64-Bit Server VM",
vm_version: "25.252-b09",
vm_vendor: "Private Build",
count: 60
},
{
version: "1.8.0_265",
vm_name: "OpenJDK 64-Bit Server VM",
vm_version: "25.265-b01",
vm_vendor: "Private Build",
count: 151
}
],
mem: {
heap_used_in_bytes: 4727781333392,
heap_max_in_bytes: 6279137591296
},
threads: 43286
},
fs: {
total_in_bytes: 1267746304614400,
free_in_bytes: 762737566871552,
available_in_bytes: 762737180995584
},
plugins: [],
network_types: {
transport_types: {
security4: 211
},
http_types: {
security4: 211
}
}
}
}

```

---

<div class="post-metadata">

**Author:** ![Nitish\_Goyal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitish_goyal/32/54988_2.png) [@Nitish\_Goyal](https://discuss.elastic.co/u/Nitish_Goyal)\
**Post date:** [October 13, 2020, 6:19am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/6 "2020-10-13T06:19:05Z")

</div>

jvm.options

```auto
-Xms28672m
-Xmx28672m

-XX:+UseG1GC
-XX:MaxGCPauseMillis=300
-XX:+DisableExplicitGC

-XX:+AlwaysPreTouch
-server
-Xss1m

-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true

-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
-XX:+HeapDumpOnOutOfMemoryError

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 13, 2020, 6:41am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/7 "2020-10-13T06:41:36Z")

</div>

It is not recommended or supported to use G1GC with Java8. If you download the latest version, which uses G1GC, you can see the following in the `jvm.options` file:

```
## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

```

In order to use G1GC I believe you should therefore upgrade your JVM.

---

<div class="post-metadata">

**Author:** ![Nitish\_Goyal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitish_goyal/32/54988_2.png) [@Nitish\_Goyal](https://discuss.elastic.co/u/Nitish_Goyal)\
**Post date:** [October 13, 2020, 10:17am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/8 "2020-10-13T10:17:07Z")

</div>

@Christian_Dahlqvist

1. Would you like to recommend anything based on cluster stats api response?

2. Upgrading each box to Java \> 10 is a tedious process for us considering it's a 200 node cluster, but we will keep this in mind and upgrade when we have the bandwidth.

3. Again asking, shall we increase the max limit size of each shard from 30GB to 50GB considering we have 10G network and recovery would be quite faster. Will be observe decrease in heap usage if we reduce number of shards from 200 per node to 160 shards per node?

4. We will add these settings into our jvm.options  
--XX:G1ReservePercent=25  
-XX:InitiatingHeapOccupancyPercent=30

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 13, 2020, 10:20am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/9 "2020-10-13T10:20:53Z")

</div>

> [@Nitish\_Goyal](#):
>
> Would you like to recommend anything based on cluster stats api response?

Not that I can see. I think upgrading to Elasticsearch 7.9 is probably the best way to address this.

> [@Nitish\_Goyal](#):
>
> Upgrading each box to Java \> 10 is a tedious process for us considering it's a 200 node cluster, but we will keep this in mind and upgrade when we have the bandwidth.

I am not sure to what extent this will help, so you may want to test first.

> [@Nitish\_Goyal](#):
>
> Again asking, shall we increase the max limit size of each shard from 30GB to 50GB considering we have 10G network and recovery would be quite faster. Will be observe decrease in heap usage if we reduce number of shards from 200 per node to 160 shards per node?

I do not think this will save you a lot of heap as you are already forcemerging down to a single segment, but can not know for sure.

> [@Nitish\_Goyal](#):
>
> We will add these settings into our jvm.options  
> --XX:G1ReservePercent=25  
> -XX:InitiatingHeapOccupancyPercent=30

That is only applicable when running G1GC, which is not recommended on Java8.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 10, 2020, 10:21am UTC](https://discuss.elastic.co/t/how-to-optimise-heap-usage-on-elasticsearch-nodes/251627/10 "2020-11-10T10:21:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.