Load data from one environment to another?

Elastic Stack Elasticsearch
1

I have a requirement to take specific ES data from one instance (permanently available) and, via an automated process, move that data to another ES instance running in a local docker container.

At the time the data is determined the local docker instance isn't running.

So I need the ability to easily save the required elasticsearch data from instance a (which is always up) and load into instance b (docker container) when the container is built.

I can do this using Elasticsearch head chrome plugin to run a query and export as json. I then have a process that loads the json at the time the docker container is started.

I'm running into a formatting problem however - if I perform a curl bulk load of the data, each "row" of data must be converted from json (as outputted by the Elasticsearch head plugin) into the correct bulk format (one line).

If I load the data via curl without the bulk option I still have to strip out the metadata (took, timed_out, shards, hits, etc) from the json file.

If the data is a small amount this isn't too bad but manually modifying the json file is too time consuming when the data is large.

Is there another way to export data from one Elasticsearch instance into a format that can easily be imported into another instance at a later time?

Any suggestions would be appreciated.

Thanks.

2

The snapshot and restore API sounds like a good fit for this.

1 Like
3

I will have to check that option out - one potential issue could be the volume of data - the docker instance is running on a single node so we wouldn't want to load all of the data from the other instance (even within an index).

Thanks for the suggestion - I'll research the API.

4

Hi,

Can I suggest to check about this thread which have I think close need:

In your case the remote reindex can be the best solution as you can filter to get only the data you need for your local docker instance and prevent importing a too big volume of data.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.