# Elasticsearch stuck applyingchanges and reconciliation ended with failed predicates

**URL:** <https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [March 31, 2021, 7:48pm UTC](https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954 "2021-03-31T19:48:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fozzylyon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fozzylyon/32/86417_2.png) [@fozzylyon](https://discuss.elastic.co/u/fozzylyon)\
**Post date:** [March 31, 2021, 7:48pm UTC](https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954/1 "2021-03-31T19:48:40Z")

</div>

ECK 1.0.1  
k8s 1.16.9

The ECK operator's been stellar. But I ran into trouble deploying node resource and count changes at the same time.

In one cluster, it seems to have worked as expected, but in another the elasticsearch object is stuck applying changes:

```auto
  NAME HEALTH NODES VERSION PHASE AGE
  es-cluster green 17 7.9.3 ApplyingChanges 321d

```

The change requested increased the node count from 17 to 23 total and changed resources on the existing 17 nodes.  
New nodes were successfully added, but there was a failed\_predicates in each reconciliation attempt:

```auto
  do_not_restart_healthy_node_if_MaxUnavailable_reached

```

And it listed all pre-existing data and master nodes as causes for failure

I was using the default Update Strategy and change budget, so it should have been able to add all new nodes immediately and terminate 1 node at a time. But it didn't attempt to terminate any existing nodes.  
And after 29 reconciliation attempts over ~60 seconds, it stopped trying.

Is there a bug or known limitation in making both changes at the same time with that update strategy?  
Is there a way to kick start the watcher again?

I've tried manually restarting nodes and it has no affect on the elasticsearch object

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![fozzylyon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fozzylyon/32/86417_2.png) [@fozzylyon](https://discuss.elastic.co/u/fozzylyon)\
**Post date:** [April 6, 2021, 8:21pm UTC](https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954/2 "2021-04-06T20:21:07Z")

</div>

We found the operator had crashed because of resources. Fixing that solved everything

---

<div class="post-metadata">

**Author:** ![Thibault\_Richard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibault_richard/32/50513_2.png) [@Thibault\_Richard](https://discuss.elastic.co/u/Thibault_Richard)\
**Post date:** [April 7, 2021, 2:56pm UTC](https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954/3 "2021-04-07T14:56:40Z")

</div>

Thanks for the update and sorry for not helping earlier.

Could you share more information about your use case? How many Elastic Stack components (Elasticsearch, Kibana, APM Server, Enterprise Search, and Beats) and how many nodes per component are managed by the ECK operator?

---

<div class="post-metadata">

**Author:** ![fozzylyon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fozzylyon/32/86417_2.png) [@fozzylyon](https://discuss.elastic.co/u/fozzylyon)\
**Post date:** [April 8, 2021, 5:04pm UTC](https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954/4 "2021-04-08T17:04:28Z")

</div>

In this specific k8s cluster, there was only an Elasticsearch component. Initially, 3 master and 14 data nodes. No other components were deployed, but the operator OOM crashed when we added 6 more data nodes

Initial resources:  
limits:  
cpu: 500m  
memory: 500Mi  
requests:  
cpu: 100m  
memory: 250Mi

Now:  
limits:  
cpu: 500m  
memory: 750Mi  
requests:  
cpu: 100m  
memory: 1500Mi

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 6, 2021, 5:04pm UTC](https://discuss.elastic.co/t/elasticsearch-stuck-applyingchanges-and-reconciliation-ended-with-failed-predicates/268954/5 "2021-05-06T17:04:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.