# Elasticsearch upgrade stuck - Skipping deletion because of migrating

**URL:** <https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [January 2, 2020, 12:23pm UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568 "2020-01-02T12:23:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 2, 2020, 12:23pm UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568/1 "2020-01-02T12:23:23Z")

</div>

Hello,

I've modified request, limits and xms xmx Java options of my cluster yaml.

The upgrade is stuck and operators logs are

> {"level":"info","ts":1577967197.4311237,"logger":"elasticsearch-controller","msg":"Updating status","iteration":5457,"namespace":"elasticsearch","es\_name":"elasticsearch"}
> 
> {"level":"info","ts":1577967197.4311762,"logger":"generic-reconciler","msg":"Aggregated reconciliation results complete","result":{"Requeue":true,"RequeueAfter":10000000000}}
> 
> {"level":"info","ts":1577967197.4311981,"logger":"elasticsearch-controller","msg":"End reconcile iteration","iteration":5457,"took":0.784774123,"namespace":"elasticsearch","es\_ame":"elasticsearch"}
> 
> {"level":"info","ts":1577967207.4313903,"logger":"elasticsearch-controller","msg":"Start reconcile iteration","iteration":5458,"namespace":"elasticsearch","es\_name":"elasticsearch"}
> 
> {"level":"info","ts":1577967207.4348965,"logger":"transport","msg":"Reconciling transport certificate secrets","namespace":"elasticsearch","es\_name":"elasticsearch"}
> 
> {"level":"info","ts":1577967208.1634543,"logger":"driver","msg":"Calculated all required changes","to\_create:":5,"to\_keep:":4,"to\_delete:":6,"namespace":"elasticsearch","es\_name":"elasticsearch"}
> 
> {"level":"info","ts":1577967208.1636055,"logger":"driver","msg":"Calculated performable changes","schedule\_for\_creation\_count":0,"schedule\_for\_deletion\_count":1,"namespace":"elasticsearch","es\_name":"elasticsearch"}
> 
> {"level":"info","ts":1577967208.2052684,"logger":"driver","msg":"Skipping deletion because of migrating data","namespace":"elasticsearch","es\_name":"elasticsearch","pod\_name":"elasticsearch-es-data-sx54vph58q"}
> 
> {"level":"info","ts":1577967208.2053738,"logger":"elasticsearch-controller","msg":"Updating status","iteration":5458,"namespace":"elasticsearch","es\_name":"elasticsearch"}

I don't know what means "Skipping deletion because of migrating"

Cluster previous state was yellow, because of one data node was "evicted" because of request threshold

Operators image is:

> [docker.elastic.co/eck/eck-operator:0.9.0](http://docker.elastic.co/eck/eck-operator:0.9.0)

---

<div class="post-metadata">

**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\
**Post date:** [January 2, 2020, 4:48pm UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568/2 "2020-01-02T16:48:35Z")

</div>

That message is because we need to restart a pod to apply the new settings, but we cannot restart it because data is currently being migrated. This behavior is much improved in the beta version of the operator and I would definitely recommend upgrading.

---

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 3, 2020, 7:19am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568/3 "2020-01-03T07:19:04Z")

</div>

I will upgrade operator when the cluster upgrade process finish. But what can I do now?

Cluster is defined with 3 data nodes, but only two are running because one of them got the "evicted" signal because go over the request. Upgrade is stuck because i need one more data node for mantain the data replication?

I also saw that I have two indices for kibana. Could be because of that?  
green open .kibana\_1 6MRg6ml7RJCRLBlb3dRw4Q 1 1 7 0 169kb 84.5kb  
green open .kibana\_2 iqNOu78jQYCf8k4prtHCMg 1 1 41 8 192.5kb 103.7kb  
green open .kibana\_task\_manager\_1 9AuHqBHoRPCdsfArt-zO\_w 1 1 2 0 13.8kb 6.9kb  
green open .kibana\_task\_manager\_2

Thanks

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [January 3, 2020, 2:04pm UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568/4 "2020-01-03T14:04:53Z")

</div>

Are you using PersistentVolumes? How many Pods are currently running in your cluster?

The outputs of the following commands (stripped out of any sensitive information) would be useful to understand what's going on:

- kubectl get elasticsearch -o json
- kubectl get pods
- GET /\_cat/shards (Elasticsearch request)

---

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 7, 2020, 8:39am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568/5 "2020-01-07T08:39:47Z")

</div>

It seems that update process it's finished. Probably only time was needed...

Thanks for support and sorry for disturb.

PD: What is the final log of the update in operator?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:39am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-stuck-skipping-deletion-because-of-migrating/213568/6 "2022-11-04T07:39:10Z")

</div>


