Regarding Elasticsearch Migration between 2 clusters having same version

Elastic Stack Elasticsearch
1

Hi there,
I am trying to migrate my existing active elasticsearch cluster to newer one using snapshot/restore. But facing some problem. On restoring the indices for the first time, it is okay. When I try to restore new snapshot the second time, it says either to delete existing indices or to close them. But, if I'll close any existing indices, how will I be able to make these 2 clusters in sync with each other? Because restoring takes time and so do snapshot.
Also, I have some doubts:

  1. Why do we need to close these existing indices even if we are not appending any new entries to the indices?
  2. What is the best approach to migrate elasticsearch cluster? [In both the clusters I'm using same version of Elasticsearch (6.4.0)]
2

Hi @shiveshabhishek,

bit more info on your use case would help to give you suitable advise.

I did something similar for our log aggregation Elastic Stack setup recently. We use Logstash to ship the data to Elasticsearch.

We use daily indices. We were able to use snapshot/restore for historic indices and for the time of the migration configured two Elasticsearch outputs in Logstash so the new data was shipped to both new and old cluster. Not sure if this approach would work for you...

3

Thanks for the response @A_B . I am using fluentd instead of logstash to ship the data to Elasticsearch but in my case, fluentd can't be configured to send data to different elasticsearch clusters/instances as it is not running in my cluster. Any other approach you are aware of.. so that I can make the 2 clusters in sync and remove the older one as soon as the newer has all the latest data.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.