# Remove/Prevent duplicates with rollover
**URL:** https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717
**Category:** Elasticsearch
**Created:** [July 27, 2022, 7:22am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717 "2022-07-27T07:22:35Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 7:22am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/1 "2022-07-27T07:22:35Z")
Hi,
One of my index has an ILM (index lifecycle management) with rollovers.
The problem is, when the index receive data and rollover at the same time, the latest data are duplicated (present in both the new and old index).
How to prevent / remove duplicates after a rollover?
Thanks for your time!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 8:48am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/2 "2022-07-27T08:48:58Z")
Are you indexing immutable data or also performing updates?
Which version of Elasticsearch are you using?
How are you indexing data into Elasticsearch?
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 8:53am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/3 "2022-07-27T08:53:25Z")
Data come from a Transform, but they are not updated (I have some bucket selector which put on hold the Transform if it's not complete). The duplicated rows are identical for the both (previous and current) indexes.
Elasticsearch version 7.17.3
Filebeat \> Elasticsearch ingest node \> Transform \> ILM
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 8:58am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/4 "2022-07-27T08:58:24Z")
So the target index the transform writes to is using rollover and is managed by ILM? If that is the case I believe [this is not supported](https://www.elastic.co/guide/en/elasticsearch/reference/8.2/transform-limitations.html#transform-data-streams-destination).
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 8:59am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/5 "2022-07-27T08:59:51Z")
So how can I store the result of my Transforms?
If I need data for 3 months, then the only way is to keep a giant hot index with everything in it?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 9:00am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/6 "2022-07-27T09:00:53Z")
Yes, I believe you would need to use a single destination index and clear data using delete by query.
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 9:03am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/7 "2022-07-27T09:03:05Z")
What is delete by query?
And is there a "add by query" where I can shift the old data to an ILM index?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 9:08am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/8 "2022-07-27T09:08:49Z")
> [@bianca6](#):
>
> What is delete by query?
> **[Delete by query API | Elasticsearch Guide \[8.2\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.2/docs-delete-by-query.html)**
> [@bianca6](#):
>
> And is there a "add by query" where I can shift the old data to an ILM index?
That would probably require you to [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/8.2/docs-reindex.html) data no longer being updated before deleting it from the original index. It may be difficult to get this consistent though.
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 9:13am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/9 "2022-07-27T09:13:07Z")
A query as "reindex data between ereyesterday 00:00:00 hour and yesterday 00:00:00 hour" and then "delete those data on the same date range" will not be consistent?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 9:16am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/10 "2022-07-27T09:16:13Z")
The two operations will take some time to run and during that time you would have duplicates, and there could be failures that need to be handled. This also assumes that you do not have any data coming in late updating any of the documents that have been transferred, and I suspect this depends on the nature and logic of the transform.
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 9:17am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/11 "2022-07-27T09:17:47Z")
Ok, thanks for the advices.
I will make some test to find the best solution for my case.
Have a nice day!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 9:20am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/12 "2022-07-27T09:20:27Z")
If this solution require you to perform a reindex as well as a delete by query for every document, would it not be better to have a single transform index with a larger number of primary shards and delete documents by delete by query once they no longer need to be retained. It would be simpler and also add less load on the cluster.
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 9:28am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/13 "2022-07-27T09:28:37Z")
But if for some reason I need to keep the data for 6 months, or a year?
Or I just want to change the template, because I have a new field or way to group the data?
I need to keep it separated.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 27, 2022, 9:33am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/14 "2022-07-27T09:33:36Z")
> [@bianca6](#):
>
> But if for some reason I need to keep the data for 6 months, or a year?
Storing data in 10 indices with 1 primary shard each or 1 single index with 10 primary shards is basically the same. Having only one index is simpler and require a lot less work and load on the cluster.
> [@bianca6](#):
>
> Or I just want to change the template, because I have a new field or way to group the data?
I am not sure how you handle changes to transforms, but do not see how switching to a different time based index is any different from starting to use a new single index. You can still query the old and new singular index through an index pattern or alias and have the same issues.
---
### Author:  [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [July 27, 2022, 9:49am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/15 "2022-07-27T09:49:19Z")
And what about the Date index name? If I change the destination (previously an index) to an ingest pipeline, does it consume a lot more resources?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 24, 2022, 9:49am UTC](https://discuss.elastic.co/t/remove-prevent-duplicates-with-rollover/310717/16 "2022-08-24T09:49:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.