# Copying Incremental Data

**URL:** <https://discuss.elastic.co/t/copying-incremental-data/232356>\
**Category:** Elasticsearch\
**Created:** [May 13, 2020, 4:19am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356 "2020-05-13T04:19:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![alajangi](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@alajangi](https://discuss.elastic.co/u/alajangi)\
**Post date:** [May 13, 2020, 4:19am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/1 "2020-05-13T04:19:23Z")

</div>

Hi,

I am using elasticsearch version 6.5.1

I have 2 indexes, say index1 and index2, index2 is reindexed from index1 with same mapping,  
new data will keep come to index1, i want to copy incremental data to index2 from index1 ? could someone help me?

Thank you,  
Kishore.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 13, 2020, 8:47am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/2 "2020-05-13T08:47:08Z")

</div>

What's the reasoning for reindexing the data if there's no mapping changes?

---

<div class="post-metadata">

**Author:** ![alajangi](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@alajangi](https://discuss.elastic.co/u/alajangi)\
**Post date:** [May 13, 2020, 5:00pm UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/3 "2020-05-13T17:00:29Z")

</div>

Index1 is using for production and index2 is reindexed for my testing purpose, i want to map more fields and ingest data on those fields, if it works then i'll do the same to production index.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 13, 2020, 10:19pm UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/4 "2020-05-13T22:19:05Z")

</div>

Ok, well you can use [CCR](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-overview.html) between clusters, but there's not anything for intra-cluster "replication" like this.

---

<div class="post-metadata">

**Author:** ![alajangi](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@alajangi](https://discuss.elastic.co/u/alajangi)\
**Post date:** [May 21, 2020, 12:20am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/5 "2020-05-21T00:20:00Z")

</div>

Hi Mark, thank you for the suggestion,

can i use range query using timestamp ? I am trying to run the following query and getting following result, but nothing updating my destination index, could you please help me what am i doing wrong here ?

```auto
POST _reindex
{
  "source": {
    "index": "source_index",
    "query": {
    "range": {
      "timestamp": {
        "gte" : "May 19th 2020, 18:23:21.317",
        "lt": "now/d"
        }
    } 
    }
  },
  "dest": {
    "index": "dest_index"
      }
}

```

## Response:

```auto
{
  "took" : 295,
  "timed_out" : false,
  "total" : 0,
  "updated" : 0,
  "created" : 0,
  "deleted" : 0,
  "batches" : 0,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : []
}

```

---

<div class="post-metadata">

**Author:** ![alajangi](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@alajangi](https://discuss.elastic.co/u/alajangi)\
**Post date:** [May 21, 2020, 12:34am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/6 "2020-05-21T00:34:19Z")

</div>

solved the issue by changing "gte" : "2020-05-19T18:23:21", and putting @ before timestamp(@timestamp), is there a way that i can automate this process? searching latest timestamp in source index and run the reindex job in cron ?  
Thank you.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 21, 2020, 12:41am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/7 "2020-05-21T00:41:55Z")

</div>

There's no cron type approach for reindex, no. You'd need to do it externally.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 18, 2020, 12:42am UTC](https://discuss.elastic.co/t/copying-incremental-data/232356/8 "2020-06-18T00:42:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.