Can we import one elastic search cluster data into another?

Elastic Stack Elasticsearch
1

We have two different clusters created on Elasticsearch server

  1. cluster 6.8
  2. cluster 7.10

Just wanted to check is there any way to import data from one cluster to another? If yes, what are the ways to do it without re-indexing?

2

snapshot

3

is this the best and recommended option for importing data from cluster 6.8 to another version 7.10?

4

I think the ranking is as follows：

  1. snapshot
  2. CCR
  3. The other mess is reindex(scroll bulk)
5

As you explicitely wrote 'without reindexing', the only way would be snapshots - you need to be sure, that the data within the snapshots however has not been written with Elasticsearch version 5.

If reindexing is an option, then you can reindex from remote to retrieve data from another cluster.

6

You can use snapshot and restore to copy indices from 6.8 to 7.10 (as long as they were created in version 6.x), but moving data the other way will require reindexing as 7.10 indices can not be read by 6.8.

7

Thanks @spinscale and @Christian_Dahlqvist for the information.
Current running Elasticsearch cluster is 6.8 and we wanted to switch to 7.10.2
before switching to 7.10.2 cluster we need all the data from 6.8 to 7.10.2 cluster.

In my case there should not be problem with snapshots option.. correct?
Is there any step by step guide/documentation around snapshots?

8

If we want to go with this option reindex from remote then where we need to exactly pass source and destination urls( diff cluster url)

9

There is an example right in the linked docs. Please take a look

10

Yes ... I was going through it . I can see source URL but not destination URL

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "username": "user",
      "password": "pass"
    },
    "index": "my-index-000001",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "my-new-index-000001"
  }
}
11

You run this on the destination cluster and it reindexes from the specified source cluster.

12

Okay .... below part is optional correct?

"query": {
      "match": {
        "test": "data"
      }
13

When we migrating indices from old cluster to new cluster then what if mapping is different from source index and destination index?

14

Then you need to reindex, as restoring a snapshot would move the whole index including mappings.

15

Thanks. I was trying this reindex option but not sure about port of source&destination cluster so getting this exception

{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[8:3] [reindex] failed to parse field [source]"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[8:3] [reindex] failed to parse field [source]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "[host] must be of the form [scheme]://[host]:[port](/[pathPrefix])? but was [https://{url}]",
      "caused_by" : {
        "type" : "u_r_i_syntax_exception",
        "reason" : "The port was not defined in the [host]: https://{url}"
      }
    }
  },
  "status" : 400
}

Not sure how to find port number .... is this mandatory??

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.