# Prevent duplicates in a data stream
**URL:** https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076
**Category:** Elasticsearch
**Tags:** datastreams
**Created:** [September 1, 2021, 4:07pm UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076 "2021-09-01T16:07:41Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@romathonat](https://discuss.elastic.co/u/romathonat)
#### Post date: [September 1, 2021, 4:07pm UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/1 "2021-09-01T16:07:41Z")
Hello,
Is there a way to prevent duplicates in a data stream ?
For a given index, specifying the \_id gives us the guarantee that there will be no duplicate with same \_id.
For data streams however, it does not work apparently.
We have a (homemade) data collector that has been launched 2 times. As our data stream rolled-over, the same data has been inserted in two different backing indices, so it is present two times in our data stream, which is a huge problem to us.
Are you aware of any solution to this problem ?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 1, 2021, 10:19pm UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/2 "2021-09-01T22:19:05Z")
I don't believe this is possible due to the way datastreams works, with the rollover, as you point out.
[Deduplication made (almost) easy, thanks to Elasticsearch's Aggregations - Spoons Elastic](https://spoon-elastic.com/all-elastic-search-post/deduplication-made-almost-easy-thanks-to-elasticsearchs-aggregations/) might be a way to clean up, but I can't help with prevention sorry.
---
### Author:  [@romathonat](https://discuss.elastic.co/u/romathonat)
#### Post date: [September 2, 2021, 6:59am UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/3 "2021-09-02T06:59:45Z")
Thank you for your response and the reading you proposed.
So it seems we have two solutions here:
- Deduplicate data as in the link provided. However in our usecase we have billions of document, I am not sur about performances in this case ? But maybe it is the "big data way" of doing things, I don't know.
- "pre-allocate document to correct index" Implement a similar process of data stream on our side. We define a pattern naming convention like "my-stream-2021-09-02". For each time window that we define (say 30 days), we create a new index from the client side, following the convention.
Now each time we want to bulk new documents, we take min and max @timestamp of this bulk, we create corresponding indices if they do not exist. Finally, we bulk data and for each one extract from its timestamp the unique corresponding indice to insert to. This way, we have the guarantee that we won't have duplicates.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 2, 2021, 7:04am UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/4 "2021-09-02T07:04:10Z")
Yep, unfortunately there's not an ideal solution for this at this point.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 2, 2021, 7:27am UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/5 "2021-09-02T07:27:21Z")
If duplication of data is a problem you may need to use standard time-based indices instead of rollover so the timestamp can be used to directly identify the index. See [this old blog post](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) for additional details. Some details are out of date but the core problem and solutions remain largely the same. You can still use the split API to adjust the shard count if some indices get too large.
---
### Author:  [@romathonat](https://discuss.elastic.co/u/romathonat)
#### Post date: [September 2, 2021, 7:40am UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/6 "2021-09-02T07:40:05Z")
Yep that seems to be the second proposed solution, we are going to apply it, thank you.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 30, 2021, 7:40am UTC](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076/7 "2021-09-30T07:40:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.