# Elasticsearch 6.6.2 constantly failing with Out Of Memory Errors
**URL:** https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669
**Category:** Elasticsearch
**Created:** [March 25, 2019, 4:35am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669 "2019-03-25T04:35:06Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 25, 2019, 4:35am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/1 "2019-03-25T04:35:06Z")
I had a 7 node cluster, with 4 data nodes and 3 master eligible nodes.
Each data node is of 64GB Ram configured with 30GB heapsize, 8 cpu and 2TB SSD Disk.
All the 4 data nodes are going down, after few hours of restart constantly, with out of memory error.
I reduced my ingestion load from 20k docs/second to 15k and 10k docs/second, but it didn't help much. I kept my search rate to 0 other than the kibana monitoring dashboard refreshing once 30s to show what is the state of my cluster on.
Following are my cluster stats
`{ "_nodes" : { "total" : 7, "successful" : 7, "failed" : 0 }, "cluster_name" : "mtr-ng1", "cluster_uuid" : "C9jjlhCmR9yeMwukeQyZZA", "timestamp" : 1553487063305, "status" : "green", "indices" : { "count" : 196, "shards" : { "total" : 1534, "primaries" : 767, "replication" : 1.0, "index" : { "shards" : { "min" : 2, "max" : 10, "avg" : 7.826530612244898 }, "primaries" : { "min" : 1, "max" : 5, "avg" : 3.913265306122449 }, "replication" : { "min" : 1.0, "max" : 1.0, "avg" : 1.0 } } }, "docs" : { "count" : 9609557961, "deleted" : 47530194 }, "store" : { "size_in_bytes" : 2525518088368 }, "fielddata" : { "memory_size_in_bytes" : 135728, "evictions" : 0 }, "query_cache" : { "memory_size_in_bytes" : 4884394, "total_count" : 120970, "hit_count" : 14068, "miss_count" : 106902, "cache_size" : 595, "cache_count" : 651, "evictions" : 56 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 20164, "memory_in_bytes" : 6746448528, "terms_memory_in_bytes" : 4794965934, "stored_fields_memory_in_bytes" : 1225705688, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 1583616, "points_memory_in_bytes" : 667648162, "doc_values_memory_in_bytes" : 56545128, "index_writer_memory_in_bytes" : 69417260, "version_map_memory_in_bytes" : 636406, "fixed_bit_set_memory_in_bytes" : 159360, "max_unsafe_auto_id_timestamp" : 1553485454779, "file_sizes" : { } } }, "nodes" : { "count" : { "total" : 7, "data" : 4, "coordinating_only" : 0, "master" : 3, "ingest" : 3 }, "versions" : ["6.6.2"], "os" : { "available_processors" : 72, "allocated_processors" : 72, "names" : [{ "name" : "Linux", "count" : 7 }], "pretty_names" : [{ "pretty_name" : "CentOS Linux 7 (Core)", "count" : 7 }], "mem" : { "total_in_bytes" : 424374198272, "free_in_bytes" : 82914578432, "used_in_bytes" : 341459619840, "free_percent" : 20, "used_percent" : 80 } }, "process" : { "cpu" : { "percent" : 0 }, "open_file_descriptors" : { "min" : 423, "max" : 3719, "avg" : 2171 } }, "jvm" : { "max_uptime_in_millis" : 320148670, "versions" : [{ "version" : "1.8.0_65", "vm_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm_version" : "25.65-b01", "vm_vendor" : "Oracle Corporation", "count" : 7 }], "mem" : { "heap_used_in_bytes" : 20870601960, "heap_max_in_bytes" : 218467926016 }, "threads" : 661 }, "fs" : { "total_in_bytes" : 7685746163712, "free_in_bytes" : 4969893056512, "available_in_bytes" : 4589853052928 }, "plugins" : [], "network_types" : { "transport_types" : { "security4" : 7 }, "http_types" : { "security4" : 7 } } } }`
Following are the snapshots of kibana monitoring dashboard, at the time the problem is happening.
     
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 25, 2019, 8:42am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/2 "2019-03-25T08:42:58Z")
The spike in heap usage, and the fact that it affects all the data nodes at the same time, suggests a sudden shift in workload. Is there an expensive search running at that time?
What do `GET _nodes/hot_threads?threads=9999` and `GET _tasks?detailed` say about the work that the cluster is doing when its heap is so high?
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 25, 2019, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/3 "2019-03-25T08:55:53Z")
Hi @DavidTurner,
There are no searches running at the time. The issue is consistently reproducible. Heap is filling up to 75% as the indexing is going on, at 75% Gc kicks in, memory consumption comes down, this repeats once or twice and third or fourth time, Gc times out and memory consumption won't come down, So it holds up at 95%+ for some time and eventually node becomes unresponsive.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 25, 2019, 8:57am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/4 "2019-03-25T08:57:03Z")
Is there a way, I can know, what is actually filling up my memory. Like if my heap is 30gb and its filled up. I want to know, which component is taking which percent of this heap.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 25, 2019, 9:01am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/5 "2019-03-25T09:01:26Z")
Both hot threads and tasks detailed are saying, bulk ingestion is the only thing thats happening in the cluster.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 25, 2019, 9:02am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/6 "2019-03-25T09:02:28Z")
What do you mean "GC times out"?
The pattern of increasing heap usage followed by a drop is [normal](https://plumbr.io/blog/memory-leaks/memory-leaks-fallacies-and-misconceptions). The graph above indicates that the second GC was successful (the heap usage seems to drop back down to 4.7GB) but then there's a spike shortly after.
If we need to go deeper than the API calls I suggested above then we will need to look at a heap dump.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 25, 2019, 9:03am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/7 "2019-03-25T09:03:40Z")
What type of data are you indexing? Are you using nested documents and/or parent child relations? Are you performing updates or just indexing new immutable documents?
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 25, 2019, 9:11am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/8 "2019-03-25T09:11:09Z")
Hi @Christian_Dahlqvist,
No nested documents or parent child relations, no update, I am just indexing new immutable documents.
@DavidTurner,
The reason I mentioned Gc times out is I could see that from the elasticsearch logs.
> [2019-03-23T16:45:40,916][WARN][o.e.m.j.JvmGcMonitorService] [es1.ng2] [gc][old][2531][52] duration [45.9s], collections [1]/[46s], total [45.9s]/[44.1m], memory [29.9gb]-\>[29.9gb]/[29.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [60.1mb]-\>[62.3mb]/[66.5mb]}{[old] [29.3gb]-\>[29.3gb]/[29.3gb]} [2019-03-23T16:45:40,916][WARN][o.e.m.j.JvmGcMonitorService] [es1.ng2] [gc][2531] overhead, spent [45.9s] collecting in the last [46s]
> [2019-03-23T16:53:58,529][ERROR][o.e.x.m.c.n.NodeStatsCollector] [es1.ng2] collector [node\_stats] timed out when collecting data
> [2019-03-23T17:31:45,405][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es1.ng2] fatal error in thread [Connection evictor], exiting
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.http.pool.AbstractConnPool.closeExpired(AbstractConnPool.java:559) ~[?:?]
> at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeExpiredConnections(PoolingHttpClientConnectionManager.java:409) ~[?:?]
> at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:67) ~[?:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_65
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 25, 2019, 9:13am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/9 "2019-03-25T09:13:19Z")
> [@Sai\_Birada](#):
>
> java.lang.OutOfMemoryError: Java heap space
If you hit an `OutOfMemoryError` then, by default, Elasticsearch will have written a heap dump. The best thing to do would be to look at this to see what was consuming all the heap.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 25, 2019, 9:15am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/10 "2019-03-25T09:15:52Z")
I got the heap dumped after the above error, Is there a way I could analyze the heap dump, to figure out what is taking up memory?.
Are there any commands in elastiscearch, which can tell me, who is taking up my 4 \* 30GB memory?.
Are there any configs in elasticsearch, which I could tune to make sure that, memory consumption is throttled and I never need to worry about elasticsearch fails with Out of memory error?.
I am ok with my queries taking more time and ingestion tasks more time, My major concern is keeping elasticsearch up and running stable all the time.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 25, 2019, 9:30am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/11 "2019-03-25T09:30:47Z")
> [@Sai\_Birada](#):
>
> I got the heap dumped after the above error, Is there a way I could analyze the heap dump, to figure out what is taking up memory?.
You can use [MAT](https://www.eclipse.org/mat/) or else you can [send them to me using this link](https://upload-staging.elstc.co/u/7b4bbba3-37a4-4761-b168-ad9923b450a4) and I will take a look.
> [@Sai\_Birada](#):
>
> Are there any commands in elastiscearch, which can tell me, who is taking up my 4 \* 30GB memory?.
The heap dump is the best bet here.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 26, 2019, 3:12am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/12 "2019-03-26T03:12:30Z")
Hi @DavidTurner,
I believe I figured out the issue here,
There is this following query, which I need to run once an hour, that's filling up my memory and bringing the cluster down,
> `{ "aggs" : { "requests" : { "terms" : { "field" : "reqtype.keyword" }, "aggs": { "clouds": { "terms": { "field": "cloudname" }, "aggs": { "orgs" : { "terms" : { "field" : "companyid", "size" : 10000 }, "aggs": { "destinations": { "terms" : { "field" : "destip", "size" : 500000 }, "aggs": { "locationids" : { "terms" : { "field" : "locationid" }, "aggs": { "locations" : { "terms" : { "field" : "location" }, "aggs" : { "active_dc" : { "terms" : { "field" : "activedc" } }, "active_latency" : { "max" : { "field" : "avg", "script" : "doc[\"dc\"].value == doc[\"activedc\"].value ? doc[\"avg\"].value : -1" } }, "optimal_dc" : { "terms": { "field" : "dc", "size" : 1, "order" : { "latency" : "asc" } }, "aggs" : { "latency" : { "max" : { "field" : "avg" } } } } } } } } } } } } } } } } }, "size": 0, "query": { "bool": { "must": [{ "query_string": { "query": "isLatency : true", "analyze_wildcard": "true", "default_field": "*" } }, { "range": { "time": { "gte": prevtime * 1000, "lte": curtime * 1000, "format": "epoch_millis" } } }], "filter": [], "should": [], "must_not": [] } } }`
is there a way I can constraint the resource utilization for this query,
Its fine that this query took one hour to finish.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 26, 2019, 6:53am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/13 "2019-03-26T06:53:27Z")
You have some large size parameters specified in this deeply nested aggregation, and this will use a lot of heap. I would recommend trying to decrease these or find a different and more efficient way to run this.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 26, 2019, 7:00am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/14 "2019-03-26T07:00:23Z")
Hi @Christian_Dahlqvist,
I need to run this query to get the value out of my data, Is there any way you can think, I can make this efficient.
I already made all the fields which are part of the queries to be keywords.
And also, my major concern is constraining the resources,
Is there a way, I can make sure, Irrespective how complex the nested aggregations are, ES will just either take hours to finish or timeout after an hour, saying that your query is beyond the capability of my resources instead of going unresponsive and down?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 26, 2019, 8:04am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/15 "2019-03-26T08:04:10Z")
> [@Sai\_Birada](#):
>
> And also, my major concern is constraining the resources,
Then you should reduce the depth of nesting in this aggregation, and reduce the sizes to something more reasonable. Do you really need the top 500000 `destip` values? Do you really need to do this for the top 10000 `companyid`s?
It looks like you're trying to find instances of high latency - is it not sufficient to look at the top few? I cannot really see how the result of this query could be useful to you without further post-processing, because it will be too large. You need to refine your query to one that gives you the information you are actually looking for.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 26, 2019, 8:21am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/16 "2019-03-26T08:21:06Z")
Might you be able to split it up into a number of smaller aggregations by using [composite aggregations](https://www.elastic.co/blog/composite-aggregations-elasticsearch-pizza-delivery-metrics), while at the same time reducing the size parameters to a lower value that still captures what you want?
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 26, 2019, 8:39am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/17 "2019-03-26T08:39:02Z")
Hi @DavidTurner,
Basically I am running this query on my raw mtr data index and reingesting the data back to an elasticsearch index mtr suggestions, which acts as my actual database. Where I can search for any company or destip and compare their active latency and their optimal latency.
If I reduce these companyid or destip numbers, I may end up loosing the information of those particular companies or destinatins in my suggestions index.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 26, 2019, 8:46am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/18 "2019-03-26T08:46:06Z")
Hi @Christian_Dahlqvist,
Composite aggregations seem to be the way for my use case, I will try it out.
But, is there no way, I could fix the resources used by searches on the ES cluster?. I don't want some terms aggregation to bring the cluster down. ES can break the query or time it out, or take huge time to finish, But becoming unresponsive will disturb the whole pipeline.
Is there no way, but to increase the heap size of the cluster?, If ever I want to run the query. (If even after optimization if it is still failing the cluster with OOM)
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 26, 2019, 8:50am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/19 "2019-03-26T08:50:53Z")
One additional thing you may want to look at is [the rollup API](https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-rollup.html)? Am not sure how this handle fields with high cardinality, but it might be worth trying. Maybe someone else with more practical experience can share some light.
> **[Data Rollups in Elasticsearch: You Know, for Saving Space
| Elastic](https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space)**
>
> This post is part of the Elastic{ON} 2018 blog series where we recap specific demos and related deep-dive sessions from the conference. From machine learning forecasting to APM to secur...
> **[How to Create, Manage, and Visualize Elasticsearch Rollup Data in Kibana](https://www.elastic.co/blog/how-to-create-manage-and-visualize-elasticsearch-rollup-data-in-kibana)**
>
> Learn how to save space by rolling up historical Elasticsearch data into summary documents that you can create, manage, and visualize with new tools in Kibana.
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [March 26, 2019, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669/20 "2019-03-26T08:55:16Z")
Hi @Christian_Dahlqvist,
Rollup API is indeed a very useful feature and reduced a great deal of work I do with scripts. I am already rolling up my indices using index management policy.
My query runs only on current days and previous day's index, the whole index size of those two indices come to 30gb, and 60gb with 1 replica, However, my heap size is around 120 Gb.
Can you direct me to some page or info, which I can look to understand, why a query on the 30GB dataset is pulling down a cluster of the 120GB heap?
[Next page](https://discuss.elastic.co/t/elasticsearch-6-6-2-constantly-failing-with-out-of-memory-errors/173669.md?page=2)