Copying Incremental Data

Elastic Stack Elasticsearch
1

Hi,

I am using elasticsearch version 6.5.1

I have 2 indexes, say index1 and index2, index2 is reindexed from index1 with same mapping,
new data will keep come to index1, i want to copy incremental data to index2 from index1 ? could someone help me?

Thank you,
Kishore.

2

What's the reasoning for reindexing the data if there's no mapping changes?

3

Index1 is using for production and index2 is reindexed for my testing purpose, i want to map more fields and ingest data on those fields, if it works then i'll do the same to production index.

4

Ok, well you can use CCR between clusters, but there's not anything for intra-cluster "replication" like this.

5

Hi Mark, thank you for the suggestion,

can i use range query using timestamp ? I am trying to run the following query and getting following result, but nothing updating my destination index, could you please help me what am i doing wrong here ?

POST _reindex
{
  "source": {
    "index": "source_index",
    "query": {
    "range": {
      "timestamp": {
        "gte" :  "May 19th 2020, 18:23:21.317",
        "lt": "now/d"
        }
    } 
    }
  },
  "dest": {
    "index": "dest_index"
      }
}

Response:

{
  "took" : 295,
  "timed_out" : false,
  "total" : 0,
  "updated" : 0,
  "created" : 0,
  "deleted" : 0,
  "batches" : 0,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}
6

solved the issue by changing "gte" : "2020-05-19T18:23:21", and putting @ before timestamp(@timestamp), is there a way that i can automate this process? searching latest timestamp in source index and run the reindex job in cron ?
Thank you.

7

There's no cron type approach for reindex, no. You'd need to do it externally.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.