Queries on setting up new AWS ES domain

Elastic Stack Elasticsearch
1

Hello there,

I'm trying to configure a new AWS ES domain for Prod and would need to migrate existing elk cluster (running on ec2 instance) logs to the new AWS ES domain. The elk cluster on Prod is currently up and running. What I'm concerned about is whether to configure the AWS ES domain first (by streaming logs using Logstash to AWS ES) OR Taking the snapshot and restoring the snapshot on AWS ES domain first

  1. If I restore the snapshot first and then configure logstash for AWS ES domain,by the time i finish restoring the setup and start configuring logstash to stream logs to AWS ES domain, I may loose some data inbetween

  2. If I configure the domain first by streaming the data from Logstash to AWS ES domain and then take snapshot of old elk cluster and restore it on new ES domain, I may get duplicate data from the snapshot to the AWS ES domain.

Can anyone please suggest me better ways to get this done so that we don't loose any index/data from old elk cluster.

Also is there a way to check and verify that all the data/log is migrated. This is to make sure that we are not loosing any logs in the new AWS ES domain as we will be shutting down the old elk cluster once this setup is complete

Thanks!

2

Set up the new cluster and then change your Logstash config so you are writing to both clusters. You can then wait for the current index to fill up in the original cluster, before creating a snapshot and migrating all the old data. Once the new cluster contains all the data you can disable the feed to the old cluster and switch over.

Before initiating the migration, make sure the new cluster has the same or higher version of Elasticsearch than the original cluster and that they are compatible.

3

Thanks for the response Christian.
Can you please elaborate "You can then wait for the current index to fill up in the original cluster"

Thanks!

4

When you start indexing into two clusters, you will have one index in the new cluster that only contains partial data 9depending on when you switch). The same index in the original cluster will however continue filling up until it is complete. Once it is deemed complete and no more data will be indexed into it, you need to move this to the new cluster and replace the partial one.

5

Hi Christian

Thank you for the response and apologies for delayed reply.

Can you please tell me how can i know that the old index is completely filled as there is continuous data feed from the servers into the old elk cluster.

Thanks!

6

That typically depends on how late data can arrive in your ingest pipeline, so it is hard for us to tell.

7

Thanks for your help Christian!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.