# ES goes to red when node restarts

**URL:** <https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716>\
**Category:** Elasticsearch\
**Created:** [August 5, 2019, 4:23am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716 "2019-08-05T04:23:02Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\
**Post date:** [August 5, 2019, 4:23am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/1 "2019-08-05T04:23:02Z")

</div>

Hi

I am using ES 6.4 version. And I have 3 machines, Each of them have one instance of ES master and ES data node running as separate processes. totally 6 nodes, 3 master and 3 data nodes. Replication Factor is being set to 1.

When we restart all nodes together, ES goes to red state. We had set min master and min data node required to be 2. Any thing else can lead to this issue?

```
metrics-2019.08.01 2 p UNASSIGNED CLUSTER_RECOVERED
metrics-2019.08.01 2 r UNASSIGNED CLUSTER_RECOVERED

{
  "index": "metrics-2019.08.01",
 "shard": 2,
 "primary": true,
  "current_state": "unassigned",
"unassigned_info": {
  "reason": "CLUSTER_RECOVERED",
"at": "2019-08-01T19:22:50.394Z",
"last_allocation_status": "no_valid_shard_copy"
},
  "can_allocate": "no_valid_shard_copy",
"allocate_explanation": "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",
"node_allocation_decisions": [
{
  "node_id": "I8hrSGVsQcO0c7DQTdmdgA",
  "node_name": "metrics-datastore-1",
  "transport_address": "192.168.25.79:9300",
  "node_attributes": {
    "ml.machine_memory": "33566429184",
    "ml.max_open_jobs": "20",
    "xpack.installed": "true",
    "ml.enabled": "true"
  },
  "node_decision": "no",
  "store": {
    "found": false
  }
},
{
  "node_id": "L-TlEqTJRjuQKJBMFsnSgw",
  "node_name": "metrics-datastore-0",
  "transport_address": "192.168.25.18:9300",
  "node_attributes": {
    "ml.machine_memory": "33566429184",
    "ml.max_open_jobs": "20",
    "xpack.installed": "true",
    "ml.enabled": "true"
  },
  "node_decision": "no",
  "store": {
    "found": false
  }
},
{
  "node_id": "zTKAccDPSZezu7iyYbVVww",
  "node_name": "metrics-datastore-2",
  "transport_address": "192.168.25.53:9300",
  "node_attributes": {
    "ml.machine_memory": "33566429184",
    "ml.max_open_jobs": "20",
    "xpack.installed": "true",
    "ml.enabled": "true"
  },
  "node_decision": "no",
  "store": {
    "found": false
  }
}
]
}
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 5, 2019, 6:12am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/2 "2019-08-05T06:12:39Z")

</div>

This response is indicating that the data for this shard is completely gone. Is every shard unassigned or is it just some of them? Can you share the output of `GET _cluster/health`? Also could you share your `elasticsearch.yml` files (use [https://gist.github.com](https://gist.github.com) if they are too large to fit here).

---

<div class="post-metadata">

**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\
**Post date:** [August 5, 2019, 9:59am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/3 "2019-08-05T09:59:54Z")

</div>

Hi only one of the index among 4 or 5 indices goes to red.

```
{
  "cluster_name" : "metrics-datastore",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 3,
"active_primary_shards" : 14,
"active_shards" : 28,
"relocating_shards" : 0,
" initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 93.33333333333333
}

```

ES config File

```
cluster.name: metrics-datastore
node.name: metrics-datastore-0
node.master: false
node.data: true
node.max_local_storage_nodes: 1
path.data: /data/elasticsearch/data,logs/elasticsearch/data 
path.logs: /logs/elasticsearch
path.repo: /cfs/data/harmony_backup/esbackup
bootstrap.memory_lock: true
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: metrics-master
discovery.zen.fd.ping_interval: 10s
discovery.zen.fd.ping_retries: 5
discovery.zen.fd.ping_timeout: 120s
gateway.recover_after_master_nodes: 2
gateway.recover_after_time: 5m
http.cors.enabled: false
indices.fielddata.cache.size: 10%
indices.memory.index_buffer_size: 30%
thread_pool.write.queue_size: 2000
network.bind_host: <IP>, _local_
network.publish_host: <IP>
logger.gateway: TRACE
gateway.expected_nodes: 6
gateway.expected_master_nodes: 3
gateway.expected_data_nodes: 3
gateway.recover_after_data_nodes: 2
```

---

<div class="post-metadata">

**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\
**Post date:** [August 5, 2019, 10:05am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/4 "2019-08-05T10:05:03Z")

</div>

index conf are as below

```
{
  "index_patterns": ["*"],
  "order": 1,
  "settings": {
"index.number_of_replicas": 1,
"index.number_of_shards": 3,
     "index.merge.scheduler.max_thread_count": 1,
     "index.refresh_interval": "30s",
     "index.translog.durability": "async",
     "index.translog.flush_threshold_size": "1g",
     "index.translog.sync_interval": "10s",
     "index.unassigned.node_left.delayed_timeout": "10m",
     "index.mapping.total_fields.limit": 3000
  }
}
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 5, 2019, 10:43am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/5 "2019-08-05T10:43:13Z")

</div>

> [@Yogesh\_BG](#):
>
> `node.max_local_storage_nodes: 1`

This is the default for this setting. Are you setting this to something other than `1` on any of your nodes?

> [@Yogesh\_BG](#):
>
> `path.data: /data/elasticsearch/data,logs/elasticsearch/data`

This tells Elasticsearch to split its data between `/data/elasticsearch/data` and `$ES_HOME/logs/elasticsearch/data`. This is a fairly unusual configuration. Why are you set up like this? It's normally better to just use a single data path. Is it possible that `$ES_HOME/logs` is being cleared out on a restart?

---

<div class="post-metadata">

**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\
**Post date:** [August 5, 2019, 12:01pm UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/6 "2019-08-05T12:01:23Z")

</div>

No we don't set below property to any other value

max\_local\_storage\_nodes

and about path, we have two disks one mounted with /data and another with \<es\_home\>/logs, and they don't get cleared

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 6, 2019, 9:11am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/7 "2019-08-06T09:11:30Z")

</div>

I don't have any further ideas. The shard data is no longer anywhere that Elasticsearch can find it, so either it's looking in the wrong places or else something else has deleted it.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 6, 2019, 9:36am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/8 "2019-08-06T09:36:24Z")

</div>

Have you specified different data paths for the master and data nodes running on the same host? If not I assume the nodes may flip directories depending on which order they come up in??

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 6, 2019, 10:23am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/9 "2019-08-06T10:23:44Z")

</div>

That was my guess too, but that doesn't happen if `node.max_local_storage_nodes: 1` on every node.

---

<div class="post-metadata">

**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\
**Post date:** [August 9, 2019, 9:04am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/10 "2019-08-09T09:04:42Z")

</div>

Hi

Thanks for helping me out, found out the issue and fixed it. It was related to data.path where we had given two paths and one of them was going corrupted because of some disk issue. We can close this topic as resolved

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 6, 2019, 9:12am UTC](https://discuss.elastic.co/t/es-goes-to-red-when-node-restarts/193716/11 "2019-09-06T09:12:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.