# GKE Upgrade and PDB

**URL:** <https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [January 20, 2020, 2:27am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661 "2020-01-20T02:27:51Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vincent\_Ngai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincent_ngai/32/43594_2.png) [@Vincent\_Ngai](https://discuss.elastic.co/u/Vincent_Ngai)\
**Post date:** [January 20, 2020, 2:27am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/1 "2020-01-20T02:27:51Z")

</div>

Hey Guys,

I am install my ECK on GKE  
as you may know GKE provide a method for cluster auto upgrade or no matter as a normal upgrade for node pool (it does 1 by 1 in each zone)

if we setup PDB , it will follow the rules as max 1 hour

However i dont know if 1 hour is enough for the replica relocate to another know

What will happen to the case and do ECK correspond react for this ?

[https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster)

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [January 20, 2020, 8:23am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/2 "2020-01-20T08:23:14Z")

</div>

Hi Vincent,

ECK already sets up a PDB with a maximum of one Pod allowed to be taken down (see [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-pod-disruption-budget.html](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-pod-disruption-budget.html)). I think that should be good enough for most cases where you don't run multiple Elasticsearch Pods per Kubernetes node.

---

<div class="post-metadata">

**Author:** ![dkow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkow/32/47340_2.png) [@dkow](https://discuss.elastic.co/u/dkow)\
**Post date:** [January 20, 2020, 8:51am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/3 "2020-01-20T08:51:17Z")

</div>

Hi @Vincent_Ngai, thanks for you message.

As to the replica relocation. If you are using network attached storage the replica relocation wouldn't normally be needed - as pod disappears and gets recreated on another node, its PV follows it and gets reattached.

If you are using local storage and it gets lost during node upgrade, after new pod is up and running ES will replicate the missing state. As to how long it can take, it depends on the state size, network performance and cluster load.

Currently, we don't have any specific recommendations around this, but I've created [https://github.com/elastic/cloud-on-k8s/issues/2448](https://github.com/elastic/cloud-on-k8s/issues/2448) to track it.

---

<div class="post-metadata">

**Author:** ![Vincent\_Ngai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincent_ngai/32/43594_2.png) [@Vincent\_Ngai](https://discuss.elastic.co/u/Vincent_Ngai)\
**Post date:** [January 20, 2020, 9:16am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/4 "2020-01-20T09:16:21Z")

</div>

Yes I know it support us to set PDB  
But as I said the gke cluster will not wait forever

Let say my cluster have 3 node( each physica node with 1 eck node) and I have 1 replica for my index and my PDB only allow 1 unavailable

If the gke got upgrade  
It will then kill 1 of node  
And ECK will start reassign the shard to another node (i suppose it will? )

In the time GKE will wait becox of PDB  
However it will only wait 1 hour

Even dont know If the migration of the shard not complete  
If over 1 hr, then  
Is that means my data will have high chance lost?

---

<div class="post-metadata">

**Author:** ![Vincent\_Ngai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincent_ngai/32/43594_2.png) [@Vincent\_Ngai](https://discuss.elastic.co/u/Vincent_Ngai)\
**Post date:** [January 20, 2020, 9:20am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/5 "2020-01-20T09:20:21Z")

</div>

I am using PD (persistent disk) in GKE  
Which is zonal disk  
Pod gone will fine be, reborn pod can monut back the disk  
However I am not sure when will ECK do the migration

---

<div class="post-metadata">

**Author:** ![dkow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkow/32/47340_2.png) [@dkow](https://discuss.elastic.co/u/dkow)\
**Post date:** [January 20, 2020, 10:17am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/6 "2020-01-20T10:17:21Z")

</div>

In this case, your cluster should be fine.

If you are using [persistent disks](https://cloud.google.com/compute/docs/disks/#pdspecs) the pod will be deleted and recreated on a different k8s node. As soon as this happens, PV on your persistent disk will be attached to the pod and ES will continue operating as normal. There is no migration to be done as data on that PD was not lost. If pod coming up takes some time, a data migration to a different pod [might start](https://www.elastic.co/guide/en/elasticsearch/reference/current/delayed-allocation.html), but as soon as new pod (with already existing PV) rejoins ES cluster, [the migration will be cancelled](https://www.elastic.co/guide/en/elasticsearch/reference/current/delayed-allocation.html#_cancellation_of_shard_relocation).

---

<div class="post-metadata">

**Author:** ![Vincent\_Ngai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincent_ngai/32/43594_2.png) [@Vincent\_Ngai](https://discuss.elastic.co/u/Vincent_Ngai)\
**Post date:** [January 20, 2020, 10:48am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/7 "2020-01-20T10:48:17Z")

</div>

Thx man really address my concern

---

<div class="post-metadata">

**Author:** ![dkow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkow/32/47340_2.png) [@dkow](https://discuss.elastic.co/u/dkow)\
**Post date:** [January 20, 2020, 11:00am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/8 "2020-01-20T11:00:53Z")

</div>

Happy to help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:41am UTC](https://discuss.elastic.co/t/gke-upgrade-and-pdb/215661/9 "2022-11-04T07:41:19Z")

</div>


