How to temporarily stop a single node for storage maintenance


I'm currently running experiments with ECK and a 3-nodes Elasticsearch cluster with 2 data/master and one master only.

The idea is that the data nodes are each on their own different storage system (this is a bare metal k8s) and I use 1 replica for all my indices.

Let's say I have to do maintenance on one of the storages, I noticed that if I just forcibly take out the storage with no action, reads can continue normally but all writes will error (node is still marked online) unless I force delete the pod that can no longer write using kubectl (writing can resume if I do as the ES node is then marked offline).

Is there a way to temporarily remove one of my data nodes so that I can take down its storage for a while? Using kubectl delete obviously causes a restart of the pod.

I thought of just editing the deployment (the Elasticsearch k8s resource) and removing the node from it. But if I re-add it will it find all its previous storage and be cool with it?

Thanks for your time,

@dkvz are you using PersistentVolumes?
I'm wondering if there's a way you could manipulate the PVC resource then delete the Pod in such a way that the corresponding Pod stays Pending until the storage gets back.