If physical mechine down, how to move the bad pod to other working node when using local-volume storageclass

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I’m using elastic-local-volume as storageclass.
I shutdown a physical mechine to simulate hardware malfunction. I found the pod can not move to other node because of volume node affinity.
Could we move it to other node manually? Or operator can move it automatically few minutes later?

2

Hey @LoadingZhang,

Local persistent volumes are bound to a particular host. If that host is down, the corresponding Pod stays Pending until the host is available again. This is by design.

If you'd like a new Pod to come up on a new host with the same PersistentVolume as the one that crashed, you'll need to use network-attached PersistentVolumes that are not bound to a particular host.
If you're OK with loosing the data of that particular Volume, then you can delete the Pod that stays in the Pending state. ECK will recreate a new one automatically. Note this may cause data loss, depending on whether you have replicas in other Elasticsearch nodes.

Note the elastic-local-volumes project is still a prototype that we do not consider production-grade. There are other local PersistentVolumes implementations out there you may also want to look at.

1 Like
3

Thanks a lot, this answer is very helpful to me.