Migrating data from one node setup to new cluster

Elastic Stack Elasticsearch
1

Recently got budget to upgrade our one node setup to a clustered, kubernetes-based system. To this end I need to migrate the existing data to the new setup. Existing node will not be part of the cluster.

How?

2

Hey!

You could use the snapshot and restore feature to migrate the data. You might need a network share or some storage that can be shared to copy this data.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Cheers!

3

You could add the new nodes to the existing cluster and move your shards to the new nodes by excluding the ip of the old nodes. Once the data is migrated you can turn off the old node.

1 Like
4

Interesting idea. Seems like this would impact newer data but not existing... I suppose one could force redundancy > 1.. this might force the new cluster to make copies of everything. Then when the original node was disconnected the cluster would copy everything again.

5

hmm not really sure if it would re-copy data..

What I would test is the following.

  • Add the new nodes to the existing cluster
  • Reconfigure logstash (if used) and other clients to use the new nodes to index / query data
  • Move the shards to the new nodes
  • Exclude the old node
  • decommission the node when all shards are moved.
  • expand the index to reflect the new nodes (ie: change the primary / replication settings)

I think that should do the trick..

6

The graceful way is shard allocation filtering, which you can use to move shards around. Include / exclude based on node attributes or the IP / hostname of the old node.

The less graceful way is to rely on replication. Though you'll need to make sure that every index has a replica configured and that all the replicas are allocated (cluster state green). Then you could just remove the one old node and the cluster will still have all the data and will also allocate all the replicas on the remaining nodes again.

PS: Don't forget to set the right value for minimum master nodes.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.