What I would like to do is rebuild this server (It's gone from Development, Testing and Production in a single stroke) whilst also upgrading elasticsearch.
I was thinking... how can I:
Build a VM(s) to Migrate all my Elasticsearch Data to a new location.
Wipe the 32 Core Server
Install Elasticsearch on the Wiped Server
Migrate the Data Back
Destroy the VM (maybe, see below)
Set up another VM as a subordinate node, there for failover and handling extra requests the main server can't? (Baring in mind the VM will be a bit lighter on the specs)
What you describe is possible with a multiple node cluster. Create some new nodes and join them to the cluster, then use allocation filtering to move all the shards to the new nodes, then decommission the old nodes. Migrating the data back is the same process in reverse. You have to be careful to keep discovery.zen.minimum_master_nodes set correctly, increasing it as you add the new nodes and then reducing it as you remove the old ones.
With a single-node cluster the same idea still sort of works, but there's no way to keep discovery.zen.minimum_master_nodes set correctly as you go from 1 node to 2 and then back to 1 again.
To use another node for resilience you would have to give it the same roles as the main node, and Elasticsearch will prefer to share the load equally between them if so, so this might cause you problems. Also a two-node cluster requires discovery.zen.minimum_master_nodes: 2 which means it is not resilient to the loss of either node. For resilience you will require three nodes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.