# Metricbeat Index turns red every hours

**URL:** <https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004>\
**Category:** Elasticsearch\
**Created:** [August 31, 2017, 1:07pm UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004 "2017-08-31T13:07:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcelwolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelwolf/32/21712_2.png) [@marcelwolf](https://discuss.elastic.co/u/marcelwolf)\
**Post date:** [August 31, 2017, 1:07pm UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/1 "2017-08-31T13:07:36Z")

</div>

I run a 2 Node elasticsearch 5.5.1 cluster that holds a metricbeat index. Every about three hour the index from metricbeat turns red.

The cluster status:

```
{
  "cluster_name": "elasticsearch",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 2,
  "number_of_data_nodes": 2,
  "active_primary_shards": 16,
  "active_shards": 31,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 2,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 93.93939393939394
}

```

The shards are uassigned:

```
metricbeat-2017.08.31 0 p UNASSIGNED                        
metricbeat-2017.08.31 0 r UNASSIGNED    

```

i created a template with one shard and one replica.

The same happens to my other indices but much later, there is not that traffic.

How can i fix this? What information do also need to help me?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 1, 2017, 2:07am UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/2 "2017-09-01T02:07:46Z")

</div>

2 nodes is bad, you have no majority there for a consensus.

What do the logs show when this state change occurs?

---

<div class="post-metadata">

**Author:** ![marcelwolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelwolf/32/21712_2.png) [@marcelwolf](https://discuss.elastic.co/u/marcelwolf)\
**Post date:** [September 1, 2017, 7:27am UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/3 "2017-09-01T07:27:13Z")

</div>

`Visualize: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}`

in the kibana ui

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 1, 2017, 7:36am UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/4 "2017-09-01T07:36:35Z")

</div>

What about in Elasticsearch?

---

<div class="post-metadata">

**Author:** ![marcelwolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelwolf/32/21712_2.png) [@marcelwolf](https://discuss.elastic.co/u/marcelwolf)\
**Post date:** [September 1, 2017, 7:38am UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/5 "2017-09-01T07:38:15Z")

</div>

```
elastic-logstash-1.1.9q2k1638sy7l@ucore02.solutions.test | [2017-08-31T23:26:27,284][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[metricbeat-2017.08.31][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[metricbeat-2017.08.31][0]] containing [94] requests]"})
elastic-logstash-1.1.9q2k1638sy7l@ucore02.solutions.test | [2017-08-31T23:26:27,284][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[metricbeat-2017.08.31][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[metricbeat-2017.08.31][0]] containing [94] requests]"})

```

The shard is not active

```
elastic-logstash-1.1.y03yuuhpi2fp@ucore01.solutions.test | [2017-09-01T08:26:08,287][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[metricbeat-2017.09.01][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[metricbeat-2017.09.01][0]] containing [125] requests]"})
elastic-logstash-1.1.y03yuuhpi2fp@ucore01.solutions.test | [2017-09-01T08:26:08,287][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}
```

---

<div class="post-metadata">

**Author:** ![marcelwolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcelwolf/32/21712_2.png) [@marcelwolf](https://discuss.elastic.co/u/marcelwolf)\
**Post date:** [September 1, 2017, 7:40am UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/6 "2017-09-01T07:40:03Z")

</div>

Maybe a usefull information: it runs in docker.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 29, 2017, 7:40am UTC](https://discuss.elastic.co/t/metricbeat-index-turns-red-every-hours/99004/7 "2017-09-29T07:40:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.