I have some transient cluster settings I want to clear out. I understand that I need to do a cluster restart for these settings to be cleared in 2.4, but I want to find a way to do this with zero downtime.
My production cluster is 30 data nodes with 3 master nodes.
Testing the process on a smaller cluster with 2 data nodes and 1 master nodes resulted in no change of the transient settings after restarting both data nodes one by one. Only when the sole master node is restarted (downtime) do the transient settings go away.
I thought I just had to restart all of my master nodes, but doing this one by one in production yielded no change to the transient settings.
Is there any way to clear the transient settings with zero downtime, or do all my master nodes have to be down at the same time? If I restart every node in the cluster one by one, including the master nodes, will that clear the transient settings?