I have one ES serve which has 1.5 billion data. Now i want copy all the data another ES server. is there any way to do that like export or data sending?
I have copied big (~200-300 million docs) indices between clusters a few times, using the Reindex API, so I would recommend using that.
With 1.5 billion documents this will take some time though - perhaps a week - but this depends on the size of the documents, the bandwidth between your clusters and the number of nodes your target cluster is running on (since each node can index documents in parallel if the shards of the target index are spread over several nodes).
To speed things up you can create the new indices without replica shards, you can add those later, since each indexing operation will have to wait for an OK from the replica shards before returning OK to the client. Without replica shards the indexing finishes as soon as the documents have been added to the primary shard. I've seen 50% reduced indexing times by removing the replica shards from an index before doing the reindex.
I would recommend having a look at the snapshot/restore feature.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.