# Cluster turns to red after reboot **URL:** \ **Category:** Elasticsearch\ **Created:** [November 27, 2018, 8:46am UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298 "2018-11-27T08:46:29Z")\ **Posts on this page:** 10\ **Page:** 2
**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\ **Post date:** [November 30, 2018, 3:02pm UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/22 "2018-11-30T15:02:36Z")
meanwhile one more thing what i can tell you when cluster is in green state and any one node goes down, then everything is fine. cluster goes to yellow state and all fine even when that node comes back and joins cluster also works fine facing this issue only when all node gets restarted at the same time ---
**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\ **Post date:** [December 4, 2018, 8:42am UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/23 "2018-12-04T08:42:09Z")
Below i shared the logs when the restart happens and before for all 6nodes [https://drive.google.com/drive/folders/1pZml3vOOMdst1OJuCyl-DxdyLe4pYPqr?usp=sharing](https://drive.google.com/drive/folders/1pZml3vOOMdst1OJuCyl-DxdyLe4pYPqr?usp=sharing) when we changed unicast address to include all 3 masternode ips, all the indices went to red state and in NODE\_LEFT state one thing what i want to tell is when the nodes restart happens the IP's of the container changes' we run as a kubernetes container ---
**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\ **Post date:** [December 4, 2018, 12:49pm UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/24 "2018-12-04T12:49:33Z")
One more thing i want to ask... Is there will be any issues when node restarts and rebalance is happening, active traffic is going like we try to write to index being recovered(i want to know from ES side, i know client will fail to write thats okay, but any problem at server side) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 4, 2018, 1:22pm UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/25 "2018-12-04T13:22:13Z")
Looking at just one unassigned shard, `[logs-2018.12.04.07][1]`, I see this: ```plaintext [2018-12-04 08:14:03,184][TRACE][gateway] [metrics-master-2] [[logs-2018.12.04.07][1], node[null], [P], v[0], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2018-12-04T08:13:54.398Z]]] on node [\{metrics-datastore-2\}\{qQ995p5ERmS0O5o7yK3VtA\}\{192.168.13.70\}\{192.168.13.70:9300\}\{max_local_storage_nodes=1, master=false\}] has version [-1] of shard\ [2018-12-04 08:14:03,184][TRACE][gateway] [metrics-master-2] [[logs-2018.12.04.07][1], node[null], [P], v[0], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2018-12-04T08:13:54.398Z]]] on node [\{metrics-datastore-1\}\{C279DcEfRDeqr2wDgJF5bQ\}\{192.168.13.214\}\{192.168.13.214:9300\}\{max_local_storage_nodes=1, master=false\}] has version [6] of shard\ [2018-12-04 08:14:03,184][TRACE][gateway] [metrics-master-2] [[logs-2018.12.04.07][1], node[null], [P], v[0], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2018-12-04T08:13:54.398Z]]] on node [\{metrics-datastore-0\}\{ZxDL21BbStCXGRD2GVieNA\}\{192.168.13.17\}\{192.168.13.17:9300\}\{max_local_storage_nodes=1, master=false\}] has version [-1] of shard\ [2018-12-04 08:14:03,184][TRACE][gateway] [metrics-master-2] [logs-2018.12.04.07][1], node[null], [P], v[0], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2018-12-04T08:13:54.398Z]] candidates for allocation: [[metrics-datastore-1] -> 6, ]\ [2018-12-04 08:14:03,184][DEBUG][gateway] [metrics-master-2] [logs-2018.12.04.07][1] found 1 allocations of [logs-2018.12.04.07][1], node[null], [P], v[0], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2018-12-04T08:13:54.398Z]], highest version: [6]\ [2018-12-04 08:14:03,184][DEBUG][gateway] [metrics-master-2] [logs-2018.12.04.07][1]: not allocating, number_of_allocated_shards_found [1]\ ``` I think this tells us that the master is looking for at least two copies of this shard, but only found one, on `metrics-datastore-1`. There should have been copies on one of other two data nodes, but it looks like they were unassigned before/during the shutdown of the cluster. I think this is one of the many resilience issues fixed in later versions, perhaps ["Make index creation resilient to index closing and full cluster crashes"](https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html#_make_index_creation_resilient_to_index_closing_and_full_cluster_crashes_status_done_v5_0_0). The fix is really to upgrade. > [@Yogesh\_BG](#): > > One more thing i want to ask... Is there will be any issues when node restarts and rebalance is happening, active traffic is going like we try to write to index being recovered(i want to know from ES side, i know client will fail to write thats okay, but any problem at server side) Numerous things have been [fixed in this area in more recent versions of Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html#_completed). ---
**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\ **Post date:** [December 4, 2018, 1:48pm UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/26 "2018-12-04T13:48:09Z")
But in my setup at beginning there always shards are being assigned and started before i restart. According to you this could happen in situation where new index being created and shards are yet to allocate. before restarting I always ensured cluster is in green state, 100% active shards number and all are in assigned state there are no unassigned shards. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 4, 2018, 2:17pm UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/27 "2018-12-04T14:17:59Z")
> [@Yogesh\_BG](#): > > But in my setup at beginning there always shards are being assigned and started before i restart. And yet according to the logs they're not there when the nodes come back after the restart. Apart from upgrading I don't really know what else to suggest. Maybe someone else can help. ---
**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\ **Post date:** [December 6, 2018, 4:48pm UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/28 "2018-12-06T16:48:25Z")
Thanks for the help with lot of patience. We are trying to figure out what could be the root cause, if we ca not fix using ES 2.4. Last question i have is: we run ES nodes as kubernetes containers. When they restart the ips of each container will change. and unicast address also will change, which we are updating as part of pod restarts. So when the ES cluster comes up it comes with new unicast ips. we made a setup in aws without using kubernetes containers, 6 node syatem, created some indices and restarted all of them together. Here IPs doesn't change they are elatic IPs. So does this may matter in any case? this is the final suspect which we have. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 7, 2018, 6:37am UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/30 "2018-12-07T06:37:11Z")
> [@Yogesh\_BG](#): > > So does this may matter in any case? this is the final suspect which we have. I believe that the addresses of the nodes don't matter to Elasticsearch here - they definitely don't matter to more recent versions. ---
**Author:** ![Yogesh\_BG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogesh_bg/32/70018_2.png) [@Yogesh\_BG](https://discuss.elastic.co/u/Yogesh_BG)\ **Post date:** [December 7, 2018, 6:57am UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/31 "2018-12-07T06:57:53Z")
Okay thank you very much ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 4, 2019, 6:57am UTC](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298/32 "2019-01-04T06:57:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/cluster-turns-to-red-after-reboot/158298.md?page=1)