# Fingerprint processor allowing duplicates **URL:** https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696 **Category:** Logstash **Created:** [September 17, 2024, 6:10pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696 "2024-09-17T18:10:15Z") **Posts on this page:** 1 **Showing post:** 5
### Author: ![mmg](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@mmg](https://discuss.elastic.co/u/mmg) #### Post date: [September 17, 2024, 8:43pm UTC](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696/5 "2024-09-17T20:43:11Z")
Yes, it is really an issue, as it's skewing metrics, and duplicates will continue to happen each time the index rolls over. It's a timing issue with the source data in SQL. The write interval is in some cases longer than the query interval. The SQL query intentionally returns the same records multiple times (with some additional each time), to ensure there's eventually a superset of the data. It's not ideal, but they're trying to take the weight of the query off the SQL database, so I've been relying on a "natural key" in the dataset to fingerprint in Elastic. --- _[View the full topic](https://discuss.elastic.co/t/fingerprint-processor-allowing-duplicates/366696)._