# ES node out of memory: Java heap space

**URL:** <https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787>\
**Category:** Elasticsearch\
**Created:** [February 1, 2021, 6:13am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787 "2021-02-01T06:13:12Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 1, 2021, 6:13am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/1 "2021-02-01T06:13:12Z")

</div>

I often find out of memory: Java heap space in some nodes in our ES cluster. I have already allocated half of the total memory of each node to heap size but I still get this error. And except heap size issue, I never find a node cpu or memory are in high loading, on the contrary, the cpu usage is lower then 10% and memory are lower than 70% most of the time. Is it normal? if not, what is the possible cause of out of heap space ?

---

<div class="post-metadata">

**Author:** ![nk1506](https://avatars.discourse-cdn.com/v4/letter/n/a88e4f/32.png) [@nk1506](https://discuss.elastic.co/u/nk1506)\
**Post date:** [February 1, 2021, 6:38am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/2 "2021-02-01T06:38:47Z")

</div>

Similar issue is happening to me too. Really waiting for some right answers.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 1, 2021, 11:16pm UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/3 "2021-02-01T23:16:30Z")

</div>

It'd be helpful if you could please post the logs from the node that OOM'd as well as the output from the `_cluster/stats?pretty&human` API.

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 7:26am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/4 "2021-02-02T07:26:50Z")

</div>

> [@warkolm](#):
>
> from the `_cluster/stats?pretty&human` API.

Thank you for your reply.  
As the out of heap space occur, it always causes the ES service to stop and cluster to collapse, I have to disable some of the log collection to stabilize the cluster.

The output of the command :

> {  
> "\_nodes": {  
> "total": 7,  
> "successful": 7,  
> "failed": 0  
> },  
> "cluster\_name": "Cluster1",  
> "cluster\_uuid": "xxxxxxxxxxxxxxxxxxxxxx",  
> "timestamp": 1612241232165,  
> "status": "yellow",  
> "indices": {  
> "count": 94,  
> "shards": {  
> "total": 349,  
> "primaries": 171,  
> "replication": 1.04093567251462,  
> "index": {  
> "shards": {  
> "min": 1,  
> "max": 9,  
> "avg": 3.7127659574468086  
> },  
> "primaries": {  
> "min": 1,  
> "max": 3,  
> "avg": 1.8191489361702127  
> },  
> "replication": {  
> "min": 0.0,  
> "max": 2.0,  
> "avg": 0.875886524822695  
> }  
> }  
> },  
> "docs": {  
> "count": 268302902,  
> "deleted": 4608438  
> },  
> "store": {  
> "size": "425.1gb",  
> "size\_in\_bytes": 456494734685  
> },  
> "fielddata": {  
> "memory\_size": "167.8kb",  
> "memory\_size\_in\_bytes": 171864,  
> "evictions": 0  
> },  
> "query\_cache": {  
> "memory\_size": "454.3mb",  
> "memory\_size\_in\_bytes": 476429169,  
> "total\_count": 2383939,  
> "hit\_count": 194135,  
> "miss\_count": 2189804,  
> "cache\_size": 30555,  
> "cache\_count": 74490,  
> "evictions": 43935  
> },  
> "completion": {  
> "size": "0b",  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 2555,  
> "memory": "40mb",  
> "memory\_in\_bytes": 41988432,  
> "terms\_memory": "20.9mb",  
> "terms\_memory\_in\_bytes": 21915712,  
> "stored\_fields\_memory": "3.7mb",  
> "stored\_fields\_memory\_in\_bytes": 3969480,  
> "term\_vectors\_memory": "0b",  
> "term\_vectors\_memory\_in\_bytes": 0,  
> "norms\_memory": "2.3mb",  
> "norms\_memory\_in\_bytes": 2467328,  
> "points\_memory": "0b",  
> "points\_memory\_in\_bytes": 0,  
> "doc\_values\_memory": "13mb",  
> "doc\_values\_memory\_in\_bytes": 13635912,  
> "index\_writer\_memory": "3.8gb",  
> "index\_writer\_memory\_in\_bytes": 4101395580,  
> "version\_map\_memory": "214mb",  
> "version\_map\_memory\_in\_bytes": 224405274,  
> "fixed\_bit\_set": "13.4mb",  
> "fixed\_bit\_set\_memory\_in\_bytes": 14150592,  
> "max\_unsafe\_auto\_id\_timestamp": 1612240341731,  
> "file\_sizes": {}  
> },  
> "mappings": {  
> "field\_types": [  
> {  
> "name": "alias",  
> "count": 3,  
> "index\_count": 1  
> },  
> {  
> "name": "binary",  
> "count": 3,  
> "index\_count": 1  
> },  
> {  
> "name": "boolean",  
> "count": 150,  
> "index\_count": 31  
> },  
> {  
> "name": "byte",  
> "count": 1,  
> "index\_count": 1  
> },  
> {  
> "name": "date",  
> "count": 260,  
> "index\_count": 61  
> },  
> {  
> "name": "double",  
> "count": 191,  
> "index\_count": 22  
> },  
> {  
> "name": "flattened",  
> "count": 1,  
> "index\_count": 1  
> },  
> {  
> "name": "float",  
> "count": 236,  
> "index\_count": 22  
> },  
> {  
> "name": "geo\_point",  
> "count": 7,  
> "index\_count": 1  
> },  
> {  
> "name": "geo\_shape",  
> "count": 1,  
> "index\_count": 1  
> },  
> {  
> "name": "half\_float",  
> "count": 77,  
> "index\_count": 21  
> },  
> {  
> "name": "integer",  
> "count": 195,  
> "index\_count": 16  
> },  
> {  
> "name": "ip",  
> "count": 21,  
> "index\_count": 1  
> },  
> {  
> "name": "keyword",  
> "count": 2686,  
> "index\_count": 62  
> },  
> {  
> "name": "long",  
> "count": 5265,  
> "index\_count": 60  
> },  
> {  
> "name": "nested",  
> "count": 61,  
> "index\_count": 20  
> },  
> {  
> "name": "object",  
> "count": 4651,  
> "index\_count": 63  
> },  
> {  
> "name": "scaled\_float",  
> "count": 112,  
> "index\_count": 1  
> },  
> {  
> "name": "short",  
> "count": 21,  
> "index\_count": 7  
> },  
> {  
> "name": "text",  
> "count": 482,  
> "index\_count": 33  
> }  
> ]  
> },  
> "analysis": {  
> "char\_filter\_types": ,  
> "tokenizer\_types": ,  
> "filter\_types": ,  
> "analyzer\_types": ,  
> "built\_in\_char\_filters": ,  
> "built\_in\_tokenizers": ,  
> "built\_in\_filters": ,  
> "built\_in\_analyzers":   
> }  
> },  
> "nodes": {  
> "count": {  
> "total": 7,  
> "coordinating\_only": 0,  
> "data": 6,  
> "ingest": 5,  
> "master": 4,  
> "ml": 4,  
> "remote\_cluster\_client": 4,  
> "transform": 4,  
> "voting\_only": 0  
> },  
> "versions": [  
> "7.9.3",  
> "7.9.2",  
> "7.8.0",  
> "7.10.0",  
> "7.9.0"  
> ],  
> "os": {  
> "available\_processors": 56,  
> "allocated\_processors": 56,  
> "names": [  
> {  
> "name": "Linux",  
> "count": 7  
> }  
> ],  
> "pretty\_names": [  
> {  
> "pretty\_name": "Ubuntu 18.04.2 LTS",  
> "count": 4  
> },  
> {  
> "pretty\_name": "Ubuntu 20.10",  
> "count": 1  
> },  
> {  
> "pretty\_name": "Ubuntu 18.10",  
> "count": 1  
> },  
> {  
> "pretty\_name": "Ubuntu 18.04.1 LTS",  
> "count": 1  
> }  
> ],  
> "mem": {  
> "total": "234.9gb",  
> "total\_in\_bytes": 252248031232,  
> "free": "32.6gb",  
> "free\_in\_bytes": 35094163456,  
> "used": "202.2gb",  
> "used\_in\_bytes": 217153867776,  
> "free\_percent": 14,  
> "used\_percent": 86  
> }  
> },

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 7:28am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/5 "2021-02-02T07:28:08Z")

</div>

> ```
> "process": {
> "cpu": {
> "percent": 101
> },
> "open_file_descriptors": {
> "min": 438,
> "max": 1202,
> "avg": 809
> }
> },
> "jvm": {
> "max_uptime": "21d",
> "max_uptime_in_millis": 1822181612,
> "versions": [
> {
> "version": "14.0.1",
> "vm_name": "OpenJDK 64-Bit Server VM",
> "vm_version": "14.0.1+7",
> "vm_vendor": "AdoptOpenJDK",
> "bundled_jdk": true,
> "using_bundled_jdk": true,
> "count": 3
> },
> {
> "version": "15",
> "vm_name": "OpenJDK 64-Bit Server VM",
> "vm_version": "15+36",
> "vm_vendor": "AdoptOpenJDK",
> "bundled_jdk": true,
> "using_bundled_jdk": true,
> "count": 1
> },
> {
> "version": "15",
> "vm_name": "OpenJDK 64-Bit Server VM",
> "vm_version": "15+36-1562",
> "vm_vendor": "Oracle Corporation",
> "bundled_jdk": true,
> "using_bundled_jdk": true,
> "count": 1
> },
> {
> "version": "15.0.1",
> "vm_name": "OpenJDK 64-Bit Server VM",
> "vm_version": "15.0.1+9",
> "vm_vendor": "AdoptOpenJDK",
> "bundled_jdk": true,
> "using_bundled_jdk": true,
> "count": 2
> }
> ],
> "mem": {
> "heap_used": "64.5gb",
> "heap_used_in_bytes": 69355816272,
> "heap_max": "116gb",
> "heap_max_in_bytes": 124554051584
> },
> "threads": 613
> },
> "fs": {
> "total": "6.1tb",
> "total_in_bytes": 6784419385344,
> "free": "3.9tb",
> "free_in_bytes": 4315239600128,
> "available": "3.6tb",
> "available_in_bytes": 4009719656448
> },
> "plugins": [],
> "network_types": {
> "transport_types": {
> "netty4": 7
> },
> "http_types": {
> "netty4": 7
> }
> },
> "discovery_types": {
> "zen": 7
> },
> "packaging_types": [
> {
> "flavor": "default",
> "type": "deb",
> "count": 7
> }
> ],
> "ingest": {
> "number_of_pipelines": 19,
> "processor_stats": {
> "conditional": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "convert": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "date": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "dot_expander": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "geoip": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "grok": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "gsub": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "json": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "remove": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "rename": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "script": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> },
> "split": {
> "count": 0,
> "failed": 0,
> "current": 0,
> "time": "0s",
> "time_in_millis": 0
> }
> }
> }
> }
> 
> ```
> 
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 2, 2021, 8:09am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/6 "2021-02-02T08:09:08Z")

</div>

> [@iammanmale](#):
>
> "versions": [  
> "7.9.3",  
> "7.9.2",  
> "7.8.0",  
> "7.10.0",  
> "7.9.0"  
> ],

This is a potentially a big problem. All nodes in the cluster should always use exactly the same version unless you are actually upgrading. Shards located on newer nodes can generally not be allocated to older nodes, so you are likely to see imbalances until this is fixed.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 2, 2021, 8:28am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/7 "2021-02-02T08:28:55Z")

</div>

I cannot reiterate Christian's comments strongly enough here.

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 8:34am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/8 "2021-02-02T08:34:16Z")

</div>

Thank you for all of your reply. One more question, we are using logstash to process log and push to ES cluster. In the cluster there are digest node, data node and master node. Should I configure the logstash to push the log to all nodes no matter what type they are or just those digest node only?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 2, 2021, 8:35am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/9 "2021-02-02T08:35:37Z")

</div>

Ingest nodes?  
For your cluster size, and having 5 of 7 nodes as ingest, I would send to those.

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 8:39am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/10 "2021-02-02T08:39:46Z")

</div>

okok...some nodes have more than one roles, for better performance, I should dedicate one node for one role only, right?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 2, 2021, 8:48am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/11 "2021-02-02T08:48:44Z")

</div>

I am not sure I understand how you have assigned roles in your cluster. Could you please elaborate?

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 9:00am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/12 "2021-02-02T09:00:12Z")

</div>

There are a range of node type in ES if I understand correctly.

> **[Node | Elasticsearch Reference \[7.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html)**

At this moment, some nodes in my cluster are master node, data node and ingest node at the same time. I am think if I should configure one node for single role only even all I have just 7 nodes

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 2, 2021, 9:04am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/13 "2021-02-02T09:04:57Z")

</div>

Do all nodes have the same hardware specification?

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 9:24am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/14 "2021-02-02T09:24:38Z")

</div>

no, some of them are I7 with 32GB ram while some are i5 with 16GB ram.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 2, 2021, 9:31am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/15 "2021-02-02T09:31:53Z")

</div>

How many do you have of each? How much storage do they have? What type of storage are they using?

Elasticsearch by default assumes all nodes are equal so unless you tier your nodes you may end up with imbalances.

---

<div class="post-metadata">

**Author:** ![iammanmale](https://avatars.discourse-cdn.com/v4/letter/i/5e9695/32.png) [@iammanmale](https://discuss.elastic.co/u/iammanmale)\
**Post date:** [February 2, 2021, 9:56am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/16 "2021-02-02T09:56:51Z")

</div>

Here is the detail of those node  
ES1: i7 32GB  
ES2: i5 48GB  
ES3: i7 32GB  
ES4: i5 16GB  
ES5: i9 48GB  
ES6: i5 32GB  
ES7: i5 32GB

All of them have at least 1TB SSD storage.  
I have assign half of the memory of each machine to java heap space.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2021, 9:56am UTC](https://discuss.elastic.co/t/es-node-out-of-memory-java-heap-space/262787/17 "2021-03-02T09:56:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.