# Cluster status turned to red after generated about 2TB indices data **URL:** \ **Category:** Elasticsearch\ **Created:** [October 10, 2018, 2:38pm UTC](https://discuss.elastic.co/t/cluster-status-turned-to-red-after-generated-about-2tb-indices-data/151866 "2018-10-10T14:38:46Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![netvmdb](https://avatars.discourse-cdn.com/v4/letter/n/b9e5f3/32.png) [@netvmdb](https://discuss.elastic.co/u/netvmdb)\ **Post date:** [October 10, 2018, 2:38pm UTC](https://discuss.elastic.co/t/cluster-status-turned-to-red-after-generated-about-2tb-indices-data/151866/1 "2018-10-10T14:38:47Z")
elasticsearch / logstash / kibana oss 6.2.4 23GB log/day. Initially the cluster was ok to ingest data, but slow on building large index. But after ingest 2TB of index data, it slows down more. Eventually cluster status turned to red. How can cluster recover from this and how to prevent this happening? Thanks. {"log":"[2018-10-09T22:13:30,824][WARN][o.e.c.s.MasterService] [7BNppEw] cluster state update task [shard-started shard id [[logstash-mail-2018.09.17][1]], allocation id [Jo\_7mNnMSWKiLzrKB3FLjA], primary term [0], message [after existing recovery][shard id [[logstash-mail-2018.09.17][1]], allocation id [Jo\_7mNnMSWKiLzrKB3FLjA], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-cron-2018.09.17][2]], allocation id [QSpAD5I2SS-KrUdxudGeig], primary term [0], message [after existing recovery][shard id [[logstash-cron-2018.09.17][2]], allocation id [QSpAD5I2SS-KrUdxudGeig], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-apache-error-2018.09.17][1]], allocation id [647TJjG4QzeLOmtOtFAHMA], primary term [0], message [after existing recovery][shard id [[logstash-apache-error-2018.09.17][1]], allocation id [647TJjG4QzeLOmtOtFAHMA], primary term [0], message [after existing recovery]]] took [39.3s] above the warn threshold of 30s\n","stream":"stdout","time":"2018-10-09T22:13:30.825547563Z"} {"log":"[2018-10-09T22:13:30,824][WARN][o.e.c.s.ClusterApplierService] [7BNppEw] cluster state applier task [apply cluster state (from master [master {7BNppEw}{7BNppEwTRNuuSREQEKewnA}{GToVKQqrTsGVoMY0GWivNQ}{192.168.144.2}{192.168.144.2:9300} committed version [98] source [shard-started shard id [[logstash-mail-2018.09.17][1]], allocation id [Jo\_7mNnMSWKiLzrKB3FLjA], primary term [0], message [after existing recovery][shard id [[logstash-mail-2018.09.17][1]], allocation id [Jo\_7mNnMSWKiLzrKB3FLjA], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-cron-2018.09.17][2]], allocation id [QSpAD5I2SS-KrUdxudGeig], primary term [0], message [after existing recovery][shard id [[logstash-cron-2018.09.17][2]], allocation id [QSpAD5I2SS-KrUdxudGeig], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-apache-error-2018.09.17][1]], allocation id [647TJjG4QzeLOmtOtFAHMA], primary term [0], message [after existing recovery][shard id [[logstash-apache-error-2018.09.17][1]], allocation id [647TJjG4QzeLOmtOtFAHMA], primary term [0], message [after existing recovery]]]])] took [39.1s] above the warn threshold of 30s\n","stream":"stdout","time":"2018-10-09T22:13:30.826204607Z"} ---
**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\ **Post date:** [October 10, 2018, 3:20pm UTC](https://discuss.elastic.co/t/cluster-status-turned-to-red-after-generated-about-2tb-indices-data/151866/2 "2018-10-10T15:20:29Z")
Hi Netvmdb, These two warning aren't related to the red status, they have to do with GC taking longer than 30 seconds. For determining why the cluster is red, I'd recommend the Cluster Allocation Explain API: [https://www.elastic.co/guide/en/elasticsearch/reference/6.4/cluster-allocation-explain.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/cluster-allocation-explain.html) When running that, it should give you a big output explaining why an unassigned shard cannot be assigned. ---
**Author:** ![netvmdb](https://avatars.discourse-cdn.com/v4/letter/n/b9e5f3/32.png) [@netvmdb](https://discuss.elastic.co/u/netvmdb)\ **Post date:** [October 10, 2018, 5:28pm UTC](https://discuss.elastic.co/t/cluster-status-turned-to-red-after-generated-about-2tb-indices-data/151866/3 "2018-10-10T17:28:31Z")
Run Allocation Explain API twice. Here are the results. How can I solve "stale or corrupt" shard and "primary shard existed but can no longer be found"? curl -XGET '[http://localhost:9200/\_cluster/allocation/explain?pretty](http://localhost:9200/_cluster/allocation/explain?pretty)' > Blockquote { "index":"logstash-mail-2018.09.22", "shard":1, "primary":true, "current\_state":"unassigned", "unassigned\_info":{ "reason":"CLUSTER\_RECOVERED", "at":"2018-10-10T15:45:21.335Z", "last\_allocation\_status":"no\_valid\_shard\_copy" }, "can\_allocate":"no\_valid\_shard\_copy", "allocate\_explanation":"cannot allocate because all found copies of the shard are either stale or corrupt", "node\_allocation\_decisions":[ { "node\_id":"T5v8cMr0QB2lUXPdZtFj4A", "node\_name":"T5v8cMr", "transport\_address":"192.168.160.5:9300", "node\_decision":"no", "store":{ "found":false } }, { "node\_id":"u7qoMkYzR1Wpieh8PWZWmw", "node\_name":"u7qoMkY", "transport\_address":"192.168.160.8:9300", "node\_decision":"no", "store":{ "in\_sync":false, "allocation\_id":"1vFOeAzoTdmcMnwPIY9uZQ" } } ] } > Blockquote > { > "index" : "logstash-message-2018.09.01", > "shard" : 3, > "primary" : true, > "current\_state" : "unassigned", > "unassigned\_info" : { > "reason" : "CLUSTER\_RECOVERED", > "at" : "2018-10-10T15:45:21.323Z", > "last\_allocation\_status" : "no\_valid\_shard\_copy" > }, > "can\_allocate" : "no\_valid\_shard\_copy", > "allocate\_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster", > "node\_allocation\_decisions" : [ > { > "node\_id" : "T5v8cMr0QB2lUXPdZtFj4A", > "node\_name" : "T5v8cMr", > "transport\_address" : "192.168.160.5:9300", > "node\_decision" : "no", > "store" : { > "found" : false > } > }, > { > "node\_id" : "u7qoMkYzR1Wpieh8PWZWmw", > "node\_name" : "u7qoMkY", > "transport\_address" : "192.168.160.8:9300", > "node\_decision" : "no", > "store" : { > "found" : false > } > } > ] > } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 7, 2018, 5:29pm UTC](https://discuss.elastic.co/t/cluster-status-turned-to-red-after-generated-about-2tb-indices-data/151866/4 "2018-11-07T17:29:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.