ECK Operator 2.8+ Shutdown API/Pre-stop Hook Script Change with many node drains question

[BenB196]

Hello,

I had a question, that I couldn't really find an answer too. In ECK Operator 2.8.0 the Shutdown API was moved to the pre-stop hook: https://github.com/elastic/cloud-on-k8s/pull/6544. The question I have and what isn't clear to me, is does this new cycle ensure that the Elasticsearch cluster never enters a red state?

Example:

If I have an EKS cluster with a node group of 3 nodes, and each node holds an Elasticsearch node for an Elasticsearch cluster.

If I want to upgrade my node group nodes, I generally just spin up a new node group, drain the old nodes, then delete the old node group. Previously, I needed to do some additional work to only drain one node at a time to prevent the cluster from going red if more than one node was not fully recovered.

Question:

With this change, will the shutdown api now correctly block the draining of a second Elasticsearch node/pod if the cluster hasn't fully recovered from the previous Elasticsearch node/pod drain/relocation?

[pebrc (Peter Brachwitz)]

@BenB196

does this new cycle ensure that the Elasticsearch cluster never enters a red state?

I am afraid it does not guarantee that. The node shutdown API in Elasticsearch is not as restrictive as to block a shutdown from proceeding even if that shutdown would lead to the cluster going red. I am aware of at least one edge case where this can happen. And quickly evicting the nodes during maintenance might be another case because during "restart" shutdowns the API does not manage shards at all. It just delays re-allocation and cleanly shuts down plugins and persistent tasks on the ES node.

The addition of the script in the pre-stop hook complements the existing mechanism in the operator (we still call the shutdown API during node orchestration from the controller)

It improves the situation pre-2.8.0 only in that respect that Elasticsearch is not going to immediately try to recover the shards on the node being shutdown but instead allows for them to come back on the new k8s nodes as long as this happens within the default restart allocation delay of 5 minutes.

[BenB196]

Thanks @pebrc for the information. The Elasticsearch issue is essentially what I was looking for. A question, I don't know if you have the answer to, but is there any background progress on that issue? It seems to have been opened for a little over a year now. Having the shutdown API be able to correctly handle node rotations more safely in an automated way would be a great benefit within a Kubernetes deployment.