How to safely do k8s node maintenance

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

In case we want to delete one k8s node, what should we do to safely delete/replace ES nodes on that k8s node.

We are on bare metal cluster, and using local disk as storage class.

From what I try, the following manual seems work. Is this the right way?

  1. cordon node
  2. delete pvc on this node
  3. drain node

Is this something that ECK operator should cover?

2

Your approach seems correct. Does it work well for you?

I think the cordon/drain phase feels out of scope for being handled by ECK.

We have an issue open in Github to facilitate this process: ECK could potentially take care of the ES Pod +PVC deletion, after safely migrating the data away.

3

this seems work, but do not recommand do it in prod.

kubectl cordon k8s-node-abc

kubectl delete pvc -es-xxx --force --grace-period=0

kubectl drain k8s-node-abc --delete-local-data --ignore-daemonsets

kubectl uncordon k8s-node-abc