We have an Elasticsearch cluster for centralised log analysis in HashiCorp Nomad cluster. We are aware that ES is a stateful application and would not benefit much from the orchestrator such as HashiCorp Nomad. But we are running there to be consistent with our other application management. Below is our current setup.
We have a terraform module to create Nomad + Consul cluster, which launches an AWS autoscaling group with min=max. We don't want to autoscale here but still use ASG for ease of integration with AWS load balancers and ease of node upgrades. In case of ES we launch 3 instances with some particular tags. And during the launch of new ES cluster in Nomad, we make use of these tags to automatically populate the
cluster.initial_master_nodes. So using this we are able to work around the limitation of bootstrapping with 7.x version which otherwise requries a hardcodes values. And each of these 3 nodes act as as master eligible and data nodes. They have an EBS volume attached which is being used as ES data directory for persitance.
Now, the concern is node upgrades. Not sure how to go about updating the nodes. I want to replace these nodes with new nodes and attach the EBS volumes there for data. Below are my specific questions.
- Once the initial cluster is formed, should I remove the
- Can I simply shut down one node, launch a new node attach the EBS volume and start ES without data loss? If yes, can I repeat the same process for update the whole cluster?
- Should I still follow the instructions in this guide for upgrades? https://www.elastic.co/guide/en/elasticsearch/reference/7.6/setup-upgrade.html
Please let me know I overlooked some document/guide. Any help is appreciated.