Migrating Elasticsearch to another Server, then back again

Elastic Stack Elasticsearch
1

Hi All,

I was hoping to get some advice, we are currently in the process of building some VMware clusters to sure up our Monitoring Environment.

Currently we are running on physical hardware:

{
  "name" : "N_Gl-Yc",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "PyFbVvHVT42jCECH4l8jFQ",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "780f8c4",
    "build_date" : "2017-04-28T17:43:27.229Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.0"
  },
  "tagline" : "You Know, for Search"
}

On a 32 Core, 64GB~ server.

What I would like to do is rebuild this server (It's gone from Development, Testing and Production in a single stroke) whilst also upgrading elasticsearch.

I was thinking... how can I:

  1. Build a VM(s) to Migrate all my Elasticsearch Data to a new location.
  2. Wipe the 32 Core Server
  3. Install Elasticsearch on the Wiped Server
  4. Migrate the Data Back
  5. Destroy the VM (maybe, see below)
  6. Set up another VM as a subordinate node, there for failover and handling extra requests the main server can't? (Baring in mind the VM will be a bit lighter on the specs)

Any advice on this is gratefully received.

2

What you describe is possible with a multiple node cluster. Create some new nodes and join them to the cluster, then use allocation filtering to move all the shards to the new nodes, then decommission the old nodes. Migrating the data back is the same process in reverse. You have to be careful to keep discovery.zen.minimum_master_nodes set correctly, increasing it as you add the new nodes and then reducing it as you remove the old ones.

With a single-node cluster the same idea still sort of works, but there's no way to keep discovery.zen.minimum_master_nodes set correctly as you go from 1 node to 2 and then back to 1 again.

To use another node for resilience you would have to give it the same roles as the main node, and Elasticsearch will prefer to share the load equally between them if so, so this might cause you problems. Also a two-node cluster requires discovery.zen.minimum_master_nodes: 2 which means it is not resilient to the loss of either node. For resilience you will require three nodes.

3

Thanks David, A 2nd VM is possible.