# Continuously copy specific documents to another index?
**URL:** https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765
**Category:** Elasticsearch
**Created:** [November 17, 2020, 10:06pm UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765 "2020-11-17T22:06:33Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Thomas\_D1](https://discuss.elastic.co/u/Thomas_D1)
#### Post date: [November 17, 2020, 10:06pm UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/1 "2020-11-17T22:06:33Z")
Hi All,
we're using Elasticsearch for storing our application logs using daily indexes, which we delete after two weeks. Now we would like to keep all logs with the loglevel "error" for another 2 months.
We thought about using a separate index for all error messages and apply the different deletion rules there.
However we don't want to add the logic regarding in which index to write ("normal" or the ones for the errors) in each of our app servers logstash configs.
Is there a way to stream / continuously copy specific documents (the ones with level "error") from one index to another?
It seems like the ReIndex API could be used for that.
- Is the right way to schedule a job which runs every 5 minutes or so with the ReIndex command?
- Does ReIndex check which documents already exist and skips those?
- If we run the script every 5 mins, we probably can also add a "query" field to limit the documents compared to the last 5 mins.
Or is there a better way? 🙂
Thanks!
Thomas
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 17, 2020, 10:26pm UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/2 "2020-11-17T22:26:19Z")
You should be able to do this with Alerting, using a [search input](https://www.elastic.co/guide/en/elasticsearch/reference/current/input-search.html) and an [index action](https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-index.html).
That way you can schedule it to run once a day, for eg.
---
### Author:  [@Thomas\_D1](https://discuss.elastic.co/u/Thomas_D1)
#### Post date: [November 18, 2020, 6:03am UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/3 "2020-11-18T06:03:09Z")
> [@warkolm](#):
>
> Alerting
Thanks for the reply! Unfortunately we only have the open source version for various reasons, is there any way to do it without the X-Pack features?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 18, 2020, 6:07am UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/4 "2020-11-18T06:07:26Z")
That'd have to be a reindex with an external trigger.
---
### Author:  [@Thomas\_D1](https://discuss.elastic.co/u/Thomas_D1)
#### Post date: [November 18, 2020, 6:46am UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/5 "2020-11-18T06:46:37Z")
Ok, so we would setup a script to run every few mins, call ReIndex with a query to only match the errors.
Two last questions:
- Will ReIndex make sure matching documents are not copied twice? Thats what I understand from the docs.
- We should probably also include the timestamp in the query to limit the results to the last X minutes (basically the script interval + a few mins).
Thanks a lot for your help!
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 18, 2020, 7:15am UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/6 "2020-11-18T07:15:33Z")
> [@Thomas\_D1](#):
>
> Will ReIndex make sure matching documents are not copied twice? Thats what I understand from the docs.
Can you link to that?
> [@Thomas\_D1](#):
>
> We should probably also include the timestamp in the query to limit the results to the last X minutes (basically the script interval + a few mins).
Yes, ideally.
---
### Author:  [@Thomas\_D1](https://discuss.elastic.co/u/Thomas_D1)
#### Post date: [November 18, 2020, 7:38am UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/7 "2020-11-18T07:38:46Z")
> **[Reindex API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)**
> Setting `version_type` to `external` causes Elasticsearch to preserve the `version` from the source, create any documents that are missing, and update any documents that have an older version in the destination than they do in the source.
I understand that this means, that documents which already exist in the target index will be updated if there is a newer version available, but not inserted twice.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 18, 2020, 8:58pm UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/8 "2020-11-18T20:58:13Z")
Yep!
---
### Author:  [@Thomas\_D1](https://discuss.elastic.co/u/Thomas_D1)
#### Post date: [November 18, 2020, 9:28pm UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/9 "2020-11-18T21:28:16Z")
I'll try it out, thanks for your help! 👍
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 16, 2020, 9:28pm UTC](https://discuss.elastic.co/t/continuously-copy-specific-documents-to-another-index/255765/10 "2020-12-16T21:28:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.