# I have got replica for each indice and i am getting Red cluster in stack monitoring Please Help

**URL:** <https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring\
**Created:** [May 5, 2020, 8:34am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101 "2020-05-05T08:34:41Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rohit\_Kumbhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_kumbhar/32/67682_2.png) [@Rohit\_Kumbhar](https://discuss.elastic.co/u/Rohit_Kumbhar)\
**Post date:** [May 5, 2020, 8:34am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/1 "2020-05-05T08:34:41Z")

</div>

Its giving me High Security Alert and "Elasticsearch cluster status is red. Allocate missing primary shards and replica shards." this message

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 5, 2020, 8:44am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/2 "2020-05-05T08:44:24Z")

</div>

Hi,  
Please go to the dev Tools in kibana (little wrench icon) and execute the following commands:

```auto
GET _cluster/health

```

and

```auto
GET _cat/indices?health=red

```

and share the ouput with us.

---

<div class="post-metadata">

**Author:** ![Rohit\_Kumbhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_kumbhar/32/67682_2.png) [@Rohit\_Kumbhar](https://discuss.elastic.co/u/Rohit_Kumbhar)\
**Post date:** [May 5, 2020, 8:46am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/3 "2020-05-05T08:46:55Z")

</div>

{  
"cluster\_name" : "elk",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 3,  
"number\_of\_data\_nodes" : 1,  
"active\_primary\_shards" : 87,  
"active\_shards" : 87,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 41,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 67.96875  
}

red open .monitoring-logstash-7-2020.05.04 iZLVxS6pS3m1qZuAUkjkMg 1 0  
red open .monitoring-kibana-7-2020.05.04 PQ4hJb0VSW6658zwBcz9BA 1 0  
red open .monitoring-es-7-2020.05.04 kQiRarFZRZehWVbVdb1WVQ 1 0  
red open .monitoring-beats-7-2020.05.04 rHFeNC7OQqyYOWv0nxHJjg 1 0

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 5, 2020, 8:50am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/4 "2020-05-05T08:50:41Z")

</div>

Ok, this 4 .monitoring indicies are the problem.  
Could you check & share the output of:

```auto
GET _cluster/allocation/explain

```

---

<div class="post-metadata">

**Author:** ![Rohit\_Kumbhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_kumbhar/32/67682_2.png) [@Rohit\_Kumbhar](https://discuss.elastic.co/u/Rohit_Kumbhar)\
**Post date:** [May 5, 2020, 8:53am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/5 "2020-05-05T08:53:28Z")

</div>

{  
"index" : "abc",  
"shard" : 0,  
"primary" : false,  
"current\_state" : "unassigned",  
"unassigned\_info" : {  
"reason" : "CLUSTER\_RECOVERED",  
"at" : "2020-05-04T11:24:47.815Z",  
"last\_allocation\_status" : "no\_attempt"  
},  
"can\_allocate" : "no",  
"allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",  
"node\_allocation\_decisions" : [  
{  
"node\_id" : "SxnRYDVNQniIgOchfr1oMw",  
"node\_name" : "data-1",  
"transport\_address" : "ip:9300",  
"node\_attributes" : {  
"ml.machine\_memory" : "16657059840",  
"ml.max\_open\_jobs" : "20",  
"xpack.installed" : "true"  
},  
"node\_decision" : "no",  
"deciders" : [  
{  
"decider" : "same\_shard",  
"decision" : "NO",  
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[siem-os-kores][0], node[SxnRYDVNQniIgOchfr1oMw], [P], s[STARTED], a[id=qkLBWfCFRaaBKNT\_LSQ2sg]]"  
}  
]  
}  
]  
}

IP an index name is renamed

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 5, 2020, 9:02am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/6 "2020-05-05T09:02:58Z")

</div>

> [@Rohit\_Kumbhar](#):
>
> "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[siem-os-kores][0], node[SxnRYDVNQniIgOchfr1oMw], [P], s[STARTED], a[id=qkLBWfCFRaaBKNT\_LSQ2sg]]"

Here you can see the cause of another "problem" your unassigned shards.  
As i can see here:

> [@Rohit\_Kumbhar](#):
>
> "number\_of\_nodes" : 3,  
> "number\_of\_data\_nodes" : 1,

you have a 3 node cluster but only 1 data node.  
A Replica Shard will always be assigned to another data node than the data node with the primary shard. If there is no other data node, the shard will not be assigned.

I dont know your infrastructure but i would recommend in a 3 node Cluster to use all 3 nodes as master and data node.

If this is not possible for you, cou can set all indicies to 0 replicas.

But for your red indicies please execute the command as follow:

```auto
GET _cluster/allocation/explain
{
  "index": ".monitoring-logstash-7-2020.05.04",
  "shard": 0,
  "primary": true
}

```

and share the ouput.

---

<div class="post-metadata">

**Author:** ![Rohit\_Kumbhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_kumbhar/32/67682_2.png) [@Rohit\_Kumbhar](https://discuss.elastic.co/u/Rohit_Kumbhar)\
**Post date:** [May 5, 2020, 9:05am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/7 "2020-05-05T09:05:04Z")

</div>

> [@KoettingSimon](#):
>
> GET \_cluster/allocation/explain { "index": ".monitoring-logstash-7-2020.05.04", "shard": 0, "primary": true }

{  
"index" : ".monitoring-logstash-7-2020.05.04",  
"shard" : 0,  
"primary" : true,  
"current\_state" : "unassigned",  
"unassigned\_info" : {  
"reason" : "INDEX\_REOPENED",  
"at" : "2020-05-05T08:46:25.733Z",  
"last\_allocation\_status" : "no\_valid\_shard\_copy"  
},  
"can\_allocate" : "no\_valid\_shard\_copy",  
"allocate\_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",  
"node\_allocation\_decisions" : [  
{  
"node\_id" : "SxnRYDVNQniIgOchfr1oMw",  
"node\_name" : "data-1",  
"transport\_address" : "ip:9300",  
"node\_attributes" : {  
"ml.machine\_memory" : "16657059840",  
"ml.max\_open\_jobs" : "20",  
"xpack.installed" : "true"  
},  
"node\_decision" : "no",  
"store" : {  
"found" : false  
}  
}  
]  
}

cou can set all indicies to 0 replicas.  
(How to set this)

Is there any threat of Data loss if do as u say by enabling all 3 nodes as data nodes??

Thanks in advance for ur quick replies

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 5, 2020, 9:09am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/8 "2020-05-05T09:09:54Z")

</div>

> [@Rohit\_Kumbhar](#):
>
> "can\_allocate" : "no\_valid\_shard\_copy",  
> "allocate\_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",

Oh, do you removed a node from your Cluster?  
This problem is described in this article: [RED Elasticsearch Cluster? Panic no longer | Elastic Blog](https://www.elastic.co/blog/red-elasticsearch-cluster-panic-no-longer)

> By the explain API telling us that there is no longer a valid shard copy for our primary shard, we know we have lost all of the nodes that held a valid shard copy. At this point, the only recourse is to wait for those nodes to come back to life and rejoin the cluster. In the odd event that all nodes holding copies of this particular shard are all permanently dead, the only recourse is to use the [reroute commands](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) to allocate an empty/stale primary shard and accept the fact that data has been lost.

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 5, 2020, 9:11am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/9 "2020-05-05T09:11:48Z")

</div>

> [@Rohit\_Kumbhar](#):
>
> cou can set all indicies to 0 replicas.  
> (How to set this)

You can use this command:

```auto
PUT /indexname/_settings
{
  "number_of_replicas": 0
}

```

replace the indexname in the command with your indexname.  
Here you can use wildcards like index-\* or just \* for all indicies

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 5, 2020, 9:14am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/10 "2020-05-05T09:14:42Z")

</div>

> [@Rohit\_Kumbhar](#):
>
> Is there any threat of Data loss if do as u say by enabling all 3 nodes as data nodes??

no, the data will be distributed to all data nodes without any data loss.

---

<div class="post-metadata">

**Author:** ![Rohit\_Kumbhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_kumbhar/32/67682_2.png) [@Rohit\_Kumbhar](https://discuss.elastic.co/u/Rohit_Kumbhar)\
**Post date:** [May 5, 2020, 10:35am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/11 "2020-05-05T10:35:13Z")

</div>

Thank you @KoettingSimon for ur Help i will get back soon once i try your options

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 2, 2020, 10:35am UTC](https://discuss.elastic.co/t/i-have-got-replica-for-each-indice-and-i-am-getting-red-cluster-in-stack-monitoring-please-help/231101/12 "2020-06-02T10:35:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.