# Query from Uptime app causes OutOfMemory exception in Elasticsearch

**URL:** <https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206>\
**Category:** Synthetics\
**Created:** [June 18, 2019, 9:01am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206 "2019-06-18T09:01:19Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mattias\_Arbin](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@Mattias\_Arbin](https://discuss.elastic.co/u/Mattias_Arbin)\
**Post date:** [June 18, 2019, 9:01am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/1 "2019-06-18T09:01:19Z")

</div>

I have a three node ELK cluster with 500M documents in 900 indices. The cluster has been running without any memory-related issues for over a year.  
I recently started using heartbeat and the Uptime app.  
Version is 6.8.0.  
I suddenly started getting OutOfMemory-exceptions that crashed one of the elasticsearch nodes. After some investigations, it is clear that this happens when you click on a link in the Error list table. See screenshot. I get a crash everytime I hit the top row in the table.

![uptime_issue_2](https://us1.discourse-cdn.com/elastic/original/3X/1/3/1315fcd39c5b28e9907e88c20b94b7a874269fe0.png)

I have heartbeat data for only 15 days. 2880 heartbeats per day in daily indices. 1 primary and one replica per index.

---

<div class="post-metadata">

**Author:** ![Mattias\_Arbin](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@Mattias\_Arbin](https://discuss.elastic.co/u/Mattias_Arbin)\
**Post date:** [June 18, 2019, 9:02am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/2 "2019-06-18T09:02:36Z")

</div>

Here is the stacktrace, that does not say a lot

```
2019-06-17T14:30:21,198][WARN][o.e.m.j.JvmGcMonitorService] [ow500logan02] [gc][884] overhead, spent [3.6s] collecting in the last [3.6s]
[2019-06-17T14:30:25,980][WARN][o.e.m.j.JvmGcMonitorService] [ow500logan02] [gc][885] overhead, spent [4.7s] collecting in the last [4.7s]
[2019-06-17T14:31:44,943][ERROR][o.e.x.m.c.n.NodeStatsCollector] [ow500logan02] collector [node_stats] timed out when collecting data
[2019-06-17T14:31:45,151][WARN][o.e.m.j.JvmGcMonitorService] [ow500logan02] [gc][886] overhead, spent [59s] collecting in the last [1.3m]
[2019-06-17T14:31:45,615][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ow500logan02] fatal error in thread [elasticsearch[ow500logan02][search][T#6]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:120) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.BigByteArray.<init>(BigByteArray.java:46) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:467) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:481) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.<init>(HyperLogLogPlusPlus.java:176) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.metrics.cardinality.InternalCardinality.doReduce(InternalCardinality.java:90) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:135) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:128) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:96) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram$Bucket.reduce(InternalAutoDateHistogram.java:131) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.reduceBuckets(InternalAutoDateHistogram.java:338) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.doReduce(InternalAutoDateHistogram.java:500) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:135) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:128) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:497) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:412) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:699) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:101) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:44) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
```

---

<div class="post-metadata">

**Author:** ![Mattias\_Arbin](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@Mattias\_Arbin](https://discuss.elastic.co/u/Mattias_Arbin)\
**Post date:** [June 18, 2019, 9:05am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/3 "2019-06-18T09:05:58Z")

</div>

Start-up log. Showing java version and JVM arguments

```
58,099][INFO][o.e.n.Node] [ow500logan02] version[6.8.0], pid[82954], build[default/rpm/65b6179/2019-05-15T20:06:13.172855Z], OS[Linux/3.10.0-
957.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-06-17T14:14:58,099][INFO][o.e.n.Node] [ow500logan02] JVM arguments [-Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+
UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encod
ing=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4
j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2270712997360736217, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/opt/logan/e
lk/elasticsearch/logs, -XX:ErrorFile=/opt/logan/elk/elasticsearch/logs/hs_err_pid%p.log, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distr
ibution.flavor=default, -Des.distribution.type=rpm]
```

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [June 18, 2019, 11:22am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/4 "2019-06-18T11:22:33Z")

</div>

Hi @Mattias_Arbin,

In provided logs its showing "Heap out of memory" error. You need to check the heap space and need to resize the parameter.

> [@](#):
>
> -XX:+HeapDumpOnOutOfMemoryError,

> [@](#):
>
> java.lang.OutOfMemoryError: Java heap space

Please find below link related to this

> **[A Heap of Trouble: Managing Elasticsearch's Managed Heap](https://www.elastic.co/blog/a-heap-of-trouble)**

Also, please read below links as you have a big size cluster environment with 900 indices.

> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![Mattias\_Arbin](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@Mattias\_Arbin](https://discuss.elastic.co/u/Mattias_Arbin)\
**Post date:** [June 18, 2019, 11:50am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/5 "2019-06-18T11:50:00Z")

</div>

Thanks for the advice.  
It is possible that increasing heap size would help, but again, this cluster has been running for months without any memory-related issues. I just find it strange that queries against the fairly small heartbeat indices would cause a general heap shortage.  
I suspect that this is more of a memory leak that consumes any available heap memory in no time.

I might try raising the heap size and see if I am right.. 😉

---

<div class="post-metadata">

**Author:** ![Mattias\_Arbin](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@Mattias\_Arbin](https://discuss.elastic.co/u/Mattias_Arbin)\
**Post date:** [June 19, 2019, 8:24am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/6 "2019-06-19T08:24:06Z")

</div>

The screenshot below pretty much explains what I mean. At 09.47 I opened the Uptime app in Kibana. CPU and heap allocation goes straight up and then the node crashes.

 ![uptime_issue_3](https://us1.discourse-cdn.com/elastic/original/3X/7/a/7a71823c86cfc9fa126410b7541af30786e5f2ee.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 13, 2019, 8:24am UTC](https://discuss.elastic.co/t/query-from-uptime-app-causes-outofmemory-exception-in-elasticsearch/186206/7 "2019-07-13T08:24:14Z")

</div>

This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.