Migrate Data from Elastic Cloud to AWS ES

Elastic Stack Elasticsearch
1

Hello,

I am trying to migrate some indices from my Elastic Cloud Cluster to AWS ES.

Since everything is on cloud i am unable to achieve it with the tutorials and demos listed out in the internet;

While there are couple methods: Snapshot Restore, Re-Index, Using Logstash (elasticsearch as input and amazon-es as outtput)... I am having hard time unable to achieve the use case using any of the method.

Can you advise the best way to do so?

Regards

Kaushik

2

I don't know what is the best for this. You should ask this on AWS forums.

Out of curiosity, why are you moving to a service with less features?

3

Thanks for you rresponse @dadoonet. It was a clients requirement. But we are not entirely moving out. We just wanted to contrast the features (like X-Pack, Reporting, accessibility) to convince the client to stick to elastic. For that we needed the same data in the amazon elasticsearch.

So, is it at least possible to dump the data onto a elasticsearch running on my local system? If so how do we do it?

Lastly, how can we create a snapshot of a specific index of an instance hosted on elastic cloud?

I request you to kindly provide me some insight please?

4

You could use a custom snapshot, at least as long as the Elasticsearch version on AWS ES is the same or newer than on the source cluster.

5

Thanks for your response @Christian_Dahlqvist :slight_smile: I tried to search through the documentation, but i did not find a procedure to do it for elastic cloud instance. I did go through the APIs and some tutorials where they could give the location or path for repositories in the elasticsearch.yml file to generate the snapshot, but what do it do in case elastic cloud?

Could you please share any links or the documentation references for taking a custom snapshot of a elastic cloud index?

6

Depending on the use case, cloud.elastic.co has Security, Monitoring, Reporting, SQL, Canvas, Logs UI, Infra UI and what is coming next :slight_smile: ...

So, is it at least possible to dump the data onto a elasticsearch running on my local system? If so how do we do it?

You can use reindex from remote I guess...

7

You should be able to create your own S3 repository and snapshot to that. Whether you can import that to AWS ES or not I do however not know. As I mentioned, versions used might be a problem. A snapshot can not be read by an older version of Elasticsearch than it was created with. If AWS ES does not support the version you are running on Elastic Elasticsearch Service or newer, I suspect reindexing from remote is the best option.

8

Yes i definitely agree elastic offers some amazing features! :sunglasses:

You mean i reindex them to my local elasticsearch using remote, then create a snapshot and then try to upload the same to an S3 and try indexing it the AWS ES? :thinking:

9

The S3 could be on my own AWS account? how do i reference it while creating the snapshot of an index in elastic cloud? I generally use the Dev Tools of Kibana to run the APIs. Could you please share any references or links about the procedure please?

I do not have a problem with that as the elastic cloud i am running is a 6.3.2 and AWS also on the same :wink:

10

I have not done it myself, so would recommend you contact support.

11

@Christian_Dahlqvist thank you for your diligence in considering my request through, I understand it is a tricky one. I will try contacting the support :slight_smile:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.