# Why does it have duplicate ID with normal logstash output and fingerprint filter?

**URL:** <https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931>\
**Category:** Logstash\
**Created:** [February 20, 2021, 3:00am UTC](https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931 "2021-02-20T03:00:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bo\_Pham\_Nguyen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bo_pham_nguyen/32/48980_2.png) [@Bo\_Pham\_Nguyen](https://discuss.elastic.co/u/Bo_Pham_Nguyen)\
**Post date:** [February 20, 2021, 3:00am UTC](https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931/1 "2021-02-20T03:00:36Z")

</div>

Hi team, This is my logstash config to get data, and I use fingerprint as \_id in order to prevent duplication from data. But it still has duplicated event as below. Any help?

```auto
filter {
        if [type] == "cve" {
        fingerprint {
                key => "123ABC"
                method => "SHA256"
                source => ["id"]
                target => "[fingerprint]"
        }
        }
}

elasticsearch {
        hosts => ["http://1x.1x.1x.2:9200"]
        index => "hc-%{+YYYY.MM.dd}"
        document_id => "%{fingerprint}"
  }

```

The first event  
"  
\_id  
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d  
\_index  
hc  
\_type  
\_doc  
author  
CVEnew  
fingerprint  
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d  
"

The second event  
"  
\_id  
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d  
\_index  
hc  
\_type  
\_doc  
author  
CVEnew  
fingerprint  
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d  
"

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f11609b13dcad6a2a266c4b97886784959c4936f.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 21, 2021, 9:47pm UTC](https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931/2 "2021-02-21T21:47:43Z")

</div>

Can you post the original document?

Please format your code/logs/config using the `</>` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2021, 9:48pm UTC](https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931/3 "2021-03-21T21:48:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.