# How to handle duplicate records in datastreams using fingerprint
**URL:** https://discuss.elastic.co/t/how-to-handle-duplicate-records-in-datastreams-using-fingerprint/384260
**Category:** Logstash
**Created:** [December 26, 2025, 9:49am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-records-in-datastreams-using-fingerprint/384260 "2025-12-26T09:49:45Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)
#### Post date: [December 26, 2025, 9:49am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-records-in-datastreams-using-fingerprint/384260/1 "2025-12-26T09:49:45Z")
Hi Team,
I am ingesting data from a Kafka topic into Elasticsearch using Logstash.
The incoming data can contain duplicates, so I am using a **fingerprint filter** on a unique business field (`seqId`) and setting it as the document `_id`.
This works correctly **within a single backing index** — duplicates are not created as long as the data goes into the same index.
However, once the **data stream rolls over to a new backing index** , I start seeing duplicate documents again, even though the `_id` generated from the fingerprint remains the same.
### Setup details:
- Ingesting data using Logstash → Elasticsearch **data stream**
- Using fingerprint filter:
- Using this in the output:
- Data stream rollover is based on time
- Same `seqId` can arrive again after rollover
Thanks In Advance.
---
### Author:  [@Tortoise](https://discuss.elastic.co/u/Tortoise)
#### Post date: [December 26, 2025, 11:12am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-records-in-datastreams-using-fingerprint/384260/2 "2025-12-26T11:12:26Z")
Hello @venkatkumar229
Data streams are designed for append-only time-series data so they are not suitable when you need global de-duplication based on \_id.
\_id uniqueness is enforced only within a single backing index so after rollover the same \_id can be indexed again in a new backing index (the same behavior seen in your case)
The same thing happens with regular indices using rollover and a write alias because Elasticsearch does not check older indices for existing \_ids.
If you require exactly one document per business key (for example seqId), the recommended approach is to use a Transform or will have to avoid rollover of index (which might not be a feasible solution).
Similar post :
> [@Fingerprint processor allowing duplicates](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/5):
>
> Yes, it is really an issue, as it's skewing metrics, and duplicates will continue to happen each time the index rolls over. It's a timing issue with the source data in SQL. The write interval is in some cases longer than the query interval. The SQL query intentionally returns the same records multiple times (with some additional each time), to ensure there's eventually a superset of the data. It's not ideal, but they're trying to take the weight of the query off the SQL database, so I've been r…
Thanks!!
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [December 26, 2025, 11:34am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-records-in-datastreams-using-fingerprint/384260/3 "2025-12-26T11:34:00Z")
As Tortoise said, data streams are append-only - approx. write once, read many times indices.
Check [a similar topic](https://discuss.elastic.co/t/prevent-duplicates-in-a-data-stream/283076), [the blog](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) and [GHub](https://github.com/elastic/elasticsearch/issues/44794).