we're planning to switch from data+master design to dedicated data and dedicated master design as a lot of people already did.
The data nodes are on physical machines, the master nodes go to docker containers. Fact: these containers can't store persistent data. The master nodes do have access to their configured data directories inside the containers, but we can't mount that directories outside, so when a container goes down the data is gone.
Now, the key question is: how bad can that be? I mean, what's the worst case that can happen, if the cluster state and whatever the master nodes save is deleted before the master nodes come up again?
Thanks and regards