We are running an ES cluster as docker containers running on top of a kubernetes cluster.
Recently we needed to restart the master node, and upon doing so lost the template we created for our indices.
I have launched the data nodes attached to persistent disks in order to allow data to be preserved in case of a crash, but haven't done the same for the ES master node since I was under the impression it is only used for routing requests to the data nodes, and does not require any persistence.
I realize now that my assumption was wrong, and that the masters do hold a certain amount of data that need to be persisted in order to survive a crash of the entire cluster and not lose any data.
My question is -
What is persisted within the ES masters?
Is there a way to configure storing all persistence data in the data nodes, allowing masters to crash and resurrect without losing any configuration?