# Nodes go out of memory in domino effect

**URL:** <https://discuss.elastic.co/t/nodes-go-out-of-memory-in-domino-effect/239489>\
**Category:** Elasticsearch\
**Created:** [July 1, 2020, 12:51pm UTC](https://discuss.elastic.co/t/nodes-go-out-of-memory-in-domino-effect/239489 "2020-07-01T12:51:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![vurus](https://avatars.discourse-cdn.com/v4/letter/v/e5b9ba/32.png) [@vurus](https://discuss.elastic.co/u/vurus)\
**Post date:** [July 1, 2020, 12:51pm UTC](https://discuss.elastic.co/t/nodes-go-out-of-memory-in-domino-effect/239489/1 "2020-07-01T12:51:04Z")

</div>

Hi fellas,

I got an elasticsearch cluster (6.4.2) with hundreds of monthly indices each of which has a varying size between 500 gigabytes and 1 terabyte consisting of 5 primary and 5 replica shards (default).

The question is;

When someone query some data ranging more than 2 years (24+ monthly indices) by the web application connected to this cluster, what i observe in the slowlogs in elasticsearch side is; same query is sent to each index that is between the specified time range, and the cluster starts to respond slower, garbage collection logs are printed and eventually nodes get OOM one by one.

Is there a solution to overcome this problem? Maybe a configuration about concurrency or memory management?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 1, 2020, 4:29pm UTC](https://discuss.elastic.co/t/nodes-go-out-of-memory-in-domino-effect/239489/2 "2020-07-01T16:29:57Z")

</div>

May be your shards are too big. 200gb per shard seems excessive to me.

What are the node specifications?

What is the output of:

```auto
GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

```

If some outputs are too big, please share them on [gist.github.com](http://gist.github.com) and link them here.

May I suggest you look at the following resources about sizing:

> **[Quantitative Cluster Sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)**

> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**

> **[Managing your black Friday logs - CloudConf.IT](https://www.slideshare.net/dadoonet/managing-your-black-friday-logs-cloudconfit-93309899)**
>
> Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your a…

[![](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c2edebd5bac194c58a5df87ede6872cb41d61a8.jpeg "Managing your Black Friday Logs, Pablo Musa, Elastic, TechSummit Amsterdam") ](https://www.youtube.com/watch?v=ilP7tG6tabI)

And [Using Rally to Get Your Elasticsearch Cluster Size Right | Elastic Videos](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 29, 2020, 4:30pm UTC](https://discuss.elastic.co/t/nodes-go-out-of-memory-in-domino-effect/239489/3 "2020-07-29T16:30:06Z")

</div>

Closed on July 29, 2020, 4:30pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.