# Fingerprint processor allowing duplicates
**URL:** https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696
**Category:** Logstash
**Created:** [September 17, 2024, 6:10pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696 "2024-09-17T18:10:15Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@mgordon](https://discuss.elastic.co/u/mgordon)
#### Post date: [September 17, 2024, 6:10pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/1 "2024-09-17T18:10:15Z")
I'm ingesting log files that have known duplicates, so have implemented a Fingerprint processor in the ingest pipeline and setting that to \_id to remove the duplicates, which works perfectly. However, when the index rolls over (with the ILM policy), and the new backing index doesn't see the duplicate \_id in the data stream, so allows the insert.
Is there a recommended approach to handle this duplication?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [September 17, 2024, 7:44pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/2 "2024-09-17T19:44:53Z")
> [@mgordon](#):
>
> Is there a recommended approach to handle this duplication?
Unfortunately no, you would need to handle this outside logstash, like making a query to get the duplicate documents and using delete\_by\_query to remove one of them.
---
### Author:  [@mmg](https://discuss.elastic.co/u/mmg)
#### Post date: [September 17, 2024, 8:11pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/3 "2024-09-17T20:11:49Z")
I was afraid of that. So my only option is find the duplicate \_id records, and delete one after it's indexed?
Is there maybe a different processor in the ingest pipeline that would identify this? Or maybe something like a unique constraint on the data stream, rather than the index?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [September 17, 2024, 8:23pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/4 "2024-09-17T20:23:27Z")
> [@mmg](#):
>
> Is there maybe a different processor in the ingest pipeline that would identify this?
There is not, the `fingerprint` processor is used to do that, but it only works when you have a duplicate `_id` in the same index, if the backing index changed, then using the same `_id` as a previous document is not seen as a duplicate in this case.
What you described is an edge case of this processor, so it won't help in this case.
> [@mmg](#):
>
> Or maybe something like a unique constraint on the data stream, rather than the index?
No, the unique constraint is the `_id` of the document, but the `_id` must be unique per index, not per data stream/alias.
The solution would be to query your Elasticsearch during ingestion to see if a document id exists or not, but this can be really expensive, I would not recommend that.
How many duplicates do you have? Is this really an issue?
---
### Author:  [@mmg](https://discuss.elastic.co/u/mmg)
#### Post date: [September 17, 2024, 8:43pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/5 "2024-09-17T20:43:11Z")
Yes, it is really an issue, as it's skewing metrics, and duplicates will continue to happen each time the index rolls over.
It's a timing issue with the source data in SQL. The write interval is in some cases longer than the query interval. The SQL query intentionally returns the same records multiple times (with some additional each time), to ensure there's eventually a superset of the data. It's not ideal, but they're trying to take the weight of the query off the SQL database, so I've been relying on a "natural key" in the dataset to fingerprint in Elastic.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 18, 2024, 1:53am UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/6 "2024-09-18T01:53:05Z")
> [@mmg](#):
>
> The SQL query intentionally returns the same records multiple times (with some additional each time), to ensure there's eventually a superset of the data.
That use case is really not a good fit for data streams, which are intended to be append-only storage.
You might do better with dated indexes, where logstash will route the duplicates using the date to the same index, and duplicate ids will result in records being overwritten.