What is the best approach for ELK stack clsuter migration to different location?

Elastic Stack Elasticsearch
1

Hi Experts,

My current ELK stack(6.3.2) setup is at abc location and now data center is migrating to another xyz location . I have 3 node cluster for ES and one kibana and 1 logstash . Now I have to rebuild the entire setup at the new location . So i will be having brand new servers , separate network etc etc.
My concern here is what will be the best approach? I can think of 2

  1. Snapshot (But because my data is live and indices are per day basis on the old cluster so how I am going to achieve this) I mean without downtime how this is possible ?
  2. Re-index API approach which I think is not possible in my case because i do not have the same network ?

Please suggest what would be the best approach . Waiting for your valuable responses .

Regards
VG

2

If you have time-based indices where new indices are created at a specific time each day you can try the following approach:

  • Update Logstash to write to both clusters in parallel.
  • Then wait until the indices that were being written to at the time of the change are no longer written to.
  • Take a full snapshot of the old cluster. This will now contain all old data, including the index that was only partially written to the new cluster.
  • Go into the new cluster and delete the partially written indices.
  • Restore all indices that do not exist in the destimation cluster from the snapshot.
  • Check that the new cluster is working and has all data.
  • Change Logstash to only write to the new cluster and remove the old cluster.

If you have time-based indices with longer time periods, e.g. weekly or monthly, or are using the rollover API the process would need to be altered.

3

This sounds really good approach as i have daily Indices thank you very much @Christian_Dahlqvist . I just need to check if I can dual feed from one LS to 2 clusters as both are on different networks. Also on the old LS server I have syslog which write data to a file then LS read it and index that data , so I need to take care of that as well. Any further suggestion on this please .

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.