# Can we avoid duplicate records with fingerprint plugin or read input only once?

**URL:** <https://discuss.elastic.co/t/can-we-avoid-duplicate-records-with-fingerprint-plugin-or-read-input-only-once/115444>\
**Category:** Logstash\
**Created:** [January 14, 2018, 1:11pm UTC](https://discuss.elastic.co/t/can-we-avoid-duplicate-records-with-fingerprint-plugin-or-read-input-only-once/115444 "2018-01-14T13:11:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\
**Post date:** [January 14, 2018, 1:11pm UTC](https://discuss.elastic.co/t/can-we-avoid-duplicate-records-with-fingerprint-plugin-or-read-input-only-once/115444/1 "2018-01-14T13:11:22Z")

</div>

Hi All

I have a input as http\_poller, pulling data from a http endpoint  
and output as a http

The configuration file is working well. But the issue i have is, since logstash runs on continuous basis it goes and insert duplicate records in output.

How can i avoid that . I need a solution wherein i can read the input only once?

My dataset is bit different, here is structure

My input data looks like this, there can 'n' number of records with same Name, so i cannot take Name or Area ID or PID as the unique identifier.

| Name | Area ID | PID | Account | Cost | Budget |
| --- | --- | --- | --- | --- | --- |
| ABC Corp | 1234 | 5678 | 20 | 345 | 340 |
| ABC Corp | 1234 | 5678 | 12 | 456 | 500 |

The above one is not a duplicate record. They are different records.But when i keep the pipeline running, it inserts duplicate records

Please let me know, how can i read data only once from input. So that i dont create duplicate records in output.

I think fingerprint a solution, but i do not get any definite example.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 11, 2018, 1:12pm UTC](https://discuss.elastic.co/t/can-we-avoid-duplicate-records-with-fingerprint-plugin-or-read-input-only-once/115444/2 "2018-02-11T13:12:01Z")

</div>

Closed on February 11, 2018, 1:12pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.