# Elasticsearch cluster going down , whenever trying to run query for more than 2 days

**URL:** <https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309>\
**Category:** Elasticsearch\
**Created:** [September 21, 2020, 6:54am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309 "2020-09-21T06:54:39Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rohitarorait82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohitarorait82/32/82981_2.png) [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\
**Post date:** [September 21, 2020, 6:54am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/1 "2020-09-21T06:54:39Z")

</div>

Hi All,

I am new to elasticsearch, so don't much idea about it's performance, but whenever I am trying to run a visualization report for more than 2 days then it is resulting in cluster going down. We have some huge load like 10 million documents per day in the impacted index

We have 6 node cluster.

i). 1 coordinating node  
ii). 3 master + data node  
iii). 2 data nodes

We have created each index with 5 shards and 1 replica.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2020, 6:57am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/2 "2020-09-21T06:57:52Z")

</div>

What is the output from `GET /_cluster/stats?human&pretty`?

---

<div class="post-metadata">

**Author:** ![rohitarorait82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohitarorait82/32/82981_2.png) [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\
**Post date:** [September 21, 2020, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/3 "2020-09-21T10:23:18Z")

</div>

Hi @warkolm : Thanks for your response, please find below output

{  
"\_nodes" : {  
"total" : 6,  
"successful" : 6,  
"failed" : 0  
},  
"cluster\_name" : "Testing",  
"cluster\_uuid" : "zckRg1AbQey2LgnX3Xe5Mg",  
"timestamp" : 1600671502433,  
"status" : "yellow",  
"indices" : {  
"count" : 47,  
"shards" : {  
"total" : 194,  
"primaries" : 99,  
"replication" : 0.9595959595959596,  
"index" : {  
"shards" : {  
"min" : 2,  
"max" : 10,  
"avg" : 4.127659574468085  
},  
"primaries" : {  
"min" : 1,  
"max" : 5,  
"avg" : 2.106382978723404  
},  
"replication" : {  
"min" : 0.6,  
"max" : 1.0,  
"avg" : 0.9829787234042554  
}  
}  
},  
"docs" : {  
"count" : 13353744729,  
"deleted" : 1358  
},  
"store" : {  
"size" : "16tb",  
"size\_in\_bytes" : 17644496139954  
},  
"fielddata" : {  
"memory\_size" : "4.8kb",  
"memory\_size\_in\_bytes" : 4976,  
"evictions" : 0  
},  
"query\_cache" : {  
"memory\_size" : "64.5kb",  
"memory\_size\_in\_bytes" : 66079,  
"total\_count" : 713,  
"hit\_count" : 297,  
"miss\_count" : 416,  
"cache\_size" : 24,  
"cache\_count" : 24,  
"evictions" : 0  
},  
"completion" : {  
"size" : "0b",  
"size\_in\_bytes" : 0  
},  
"segments" : {  
"count" : 6667,  
"memory" : "184.6mb",  
"memory\_in\_bytes" : 193630968,  
"terms\_memory" : "85.5mb",  
"terms\_memory\_in\_bytes" : 89746416,  
"stored\_fields\_memory" : "78.3mb",  
"stored\_fields\_memory\_in\_bytes" : 82169848,  
"term\_vectors\_memory" : "0b",  
"term\_vectors\_memory\_in\_bytes" : 0,  
"norms\_memory" : "11mb",  
"norms\_memory\_in\_bytes" : 11582400,  
"points\_memory" : "0b",  
"points\_memory\_in\_bytes" : 0,  
"doc\_values\_memory" : "9.6mb",  
"doc\_values\_memory\_in\_bytes" : 10132304,  
"index\_writer\_memory" : "490.9mb",  
"index\_writer\_memory\_in\_bytes" : 514779868,  
"version\_map\_memory" : "37.7kb",  
"version\_map\_memory\_in\_bytes" : 38686,  
"fixed\_bit\_set" : "100.2kb",  
"fixed\_bit\_set\_memory\_in\_bytes" : 102640,  
"max\_unsafe\_auto\_id\_timestamp" : 1600671102338,  
"file\_sizes" : { }  
},  
"mappings" : {  
"field\_types" : [  
{  
"name" : "alias",  
"count" : 6,  
"index\_count" : 2  
},  
{  
"name" : "binary",  
"count" : 7,  
"index\_count" : 2  
},  
{  
"name" : "boolean",  
"count" : 211,  
"index\_count" : 31  
},  
{  
"name" : "byte",  
"count" : 10,  
"index\_count" : 10  
},  
{  
"name" : "date",  
"count" : 259,  
"index\_count" : 45  
},  
{  
"name" : "double",  
"count" : 203,  
"index\_count" : 3  
},  
{  
"name" : "flattened",  
"count" : 1,  
"index\_count" : 1  
},  
{  
"name" : "float",  
"count" : 285,  
"index\_count" : 13  
},  
{  
"name" : "geo\_point",  
"count" : 14,  
"index\_count" : 2  
},  
{  
"name" : "geo\_shape",  
"count" : 1,  
"index\_count" : 1  
},  
{  
"name" : "half\_float",  
"count" : 56,  
"index\_count" : 14  
},  
{  
"name" : "integer",  
"count" : 181,  
"index\_count" : 10  
},  
{  
"name" : "ip",  
"count" : 42,  
"index\_count" : 2  
},  
{  
"name" : "keyword",  
"count" : 2849,  
"index\_count" : 46  
},  
{  
"name" : "long",  
"count" : 5826,  
"index\_count" : 32  
},  
{  
"name" : "nested",  
"count" : 29,  
"index\_count" : 9  
},  
{  
"name" : "object",  
"count" : 5816,  
"index\_count" : 37  
},  
{  
"name" : "scaled\_float",  
"count" : 224,  
"index\_count" : 2  
},  
{  
"name" : "short",  
"count" : 16,  
"index\_count" : 8  
},  
{  
"name" : "text",  
"count" : 459,  
"index\_count" : 34  
}  
]  
},  
"analysis" : {  
"char\_filter\_types" : ,  
"tokenizer\_types" : ,  
"filter\_types" : [  
{  
"name" : "pattern\_capture",  
"count" : 1,  
"index\_count" : 1  
}  
],  
"analyzer\_types" : [  
{  
"name" : "custom",  
"count" : 1,  
"index\_count" : 1  
}  
],  
"built\_in\_char\_filters" : ,  
"built\_in\_tokenizers" : [  
{  
"name" : "uax\_url\_email",  
"count" : 1,  
"index\_count" : 1  
}  
],  
"built\_in\_filters" : [  
{  
"name" : "lowercase",  
"count" : 1,  
"index\_count" : 1  
},  
{  
"name" : "unique",  
"count" : 1,  
"index\_count" : 1  
}  
],  
"built\_in\_analyzers" :   
}  
},  
"nodes" : {  
"count" : {  
"total" : 6,  
"coordinating\_only" : 0,  
"data" : 5,  
"ingest" : 5,  
"master" : 3,  
"ml" : 6,  
"remote\_cluster\_client" : 6,  
"transform" : 5,  
"voting\_only" : 0  
},  
"versions" : [  
"7.7.0"  
],  
"os" : {  
"available\_processors" : 48,  
"allocated\_processors" : 48,  
"names" : [  
{  
"name" : "Linux",  
"count" : 6  
}  
],  
"pretty\_names" : [  
{  
"pretty\_name" : "Red Hat Enterprise Linux Server 7.8 (Maipo)",  
"count" : 6  
}  
],  
"mem" : {  
"total" : "376.5gb",  
"total\_in\_bytes" : 404268306432,  
"free" : "39.3gb",  
"free\_in\_bytes" : 42282094592,  
"used" : "337.1gb",  
"used\_in\_bytes" : 361986211840,  
"free\_percent" : 10,  
"used\_percent" : 90  
}  
},  
"process" : {  
"cpu" : {  
"percent" : 201  
},  
"open\_file\_descriptors" : {  
"min" : 565,  
"max" : 1265,  
"avg" : 1026  
}  
},  
"jvm" : {  
"max\_uptime" : "2.7d",  
"max\_uptime\_in\_millis" : 240952954,  
"versions" : [  
{  
"version" : "14",  
"vm\_name" : "OpenJDK 64-Bit Server VM",  
"vm\_version" : "14+36",  
"vm\_vendor" : "AdoptOpenJDK",  
"bundled\_jdk" : true,  
"using\_bundled\_jdk" : true,  
"count" : 6  
}  
],  
"mem" : {  
"heap\_used" : "3.6gb",  
"heap\_used\_in\_bytes" : 3931130080,  
"heap\_max" : "6gb",  
"heap\_max\_in\_bytes" : 6442450944  
},  
"threads" : 660  
},  
"fs" : {  
"total" : "41.8tb",  
"total\_in\_bytes" : 46054047244288,  
"free" : "23.6tb",  
"free\_in\_bytes" : 26053512101888,  
"available" : "23.6tb",  
"available\_in\_bytes" : 26053512101888  
},  
"plugins" : [  
{  
"name" : "mapper-size",  
"version" : "7.7.0",  
"elasticsearch\_version" : "7.7.0",  
"java\_version" : "1.8",  
"description" : "The Mapper Size plugin allows document to record their uncompressed size at index time.",  
"classname" : "org.elasticsearch.plugin.mapper.MapperSizePlugin",  
"extended\_plugins" : ,  
"has\_native\_controller" : false  
}  
],  
"network\_types" : {  
"transport\_types" : {  
"security4" : 6  
},  
"http\_types" : {  
"security4" : 6  
}  
},  
"discovery\_types" : {  
"zen" : 6  
},  
"packaging\_types" : [  
{  
"flavor" : "default",  
"type" : "rpm",  
"count" : 6  
}  
],  
"ingest" : {  
"number\_of\_pipelines" : 2,  
"processor\_stats" : {  
"gsub" : {  
"count" : 0,  
"failed" : 0,  
"current" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0  
},  
"script" : {  
"count" : 0,  
"failed" : 0,  
"current" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2020, 10:30am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/4 "2020-09-21T10:30:30Z")

</div>

Are you running with a heap size of 1GB per node with that amount of data in the cluster? If so I suspect you need to increase it significantly.

---

<div class="post-metadata">

**Author:** ![rohitarorait82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohitarorait82/32/82981_2.png) [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\
**Post date:** [September 21, 2020, 10:50am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/5 "2020-09-21T10:50:46Z")

</div>

@Christian_Dahlqvist : Can you suggest , how much heap size we need,

PS: This data is of just one index , total data is around 20 million documents per day

---

<div class="post-metadata">

**Author:** ![rohitarorait82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohitarorait82/32/82981_2.png) [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\
**Post date:** [September 21, 2020, 1:46pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/6 "2020-09-21T13:46:15Z")

</div>

@warkolm : Can you please help here.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2020, 1:52pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/7 "2020-09-21T13:52:20Z")

</div>

Set it to 50% if available RAM (or max 30GB) and see if that helps.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2020, 1:52pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-going-down-whenever-trying-to-run-query-for-more-than-2-days/249309/8 "2020-10-19T13:52:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.