I currently have a cluster that has dedicated data nodes that use PVC for storage. I am testing various configurations and as it stands now when I make a change using the manifest, the controller will take down each node one-by-one, keeping the cluster in a yellow / green state. However, due to the high number of nodes (and data stored within) this will take up to a day.
I'm curious if I can make the configuration change and then simply delete each of the data pods explicitly, accepting the fact that the cluster will enter a red state temporarily but eventually come back up and re-sync. This will be a much faster operation overall as the nodes can each come back up in parallel, accepting the fact that the state will go red until it is resync'd.