# Logstash riddle fingerprint duplicates

**URL:** <https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712>\
**Category:** Logstash\
**Created:** [December 9, 2024, 3:38pm UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712 "2024-12-09T15:38:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [December 9, 2024, 3:38pm UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712/1 "2024-12-09T15:38:58Z")

</div>

Hi there,

Using the logstash fingerprint plugin i came across an issue where it seems i cannot define the \_id as a target from the plugin using the following :

```auto
fingerprint {
    source => ["[host][name]", "[record][id]"]  
    target => "[@metadata][_id]"            
    method => "SHA256"                                        
    concatenate_sources => true             
  }

```

The configuration above strangely does not work as intented.

Do i need to ` document_id => "%{[@metadata][_id]}"` in my elastic outptut ?

Why does it feels redudant and useless while using \_id as a target ?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 9, 2024, 5:29pm UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712/2 "2024-12-09T17:29:56Z")

</div>

> [@grumo35](#):
>
> Do i need to ` document_id => "%{[@metadata][_id]}"` in my elastic outptut ?

If you want to use custom ids, then yes, you need to use the `document_id` in your Elasticsearch output.

> [@grumo35](#):
>
> Why does it feels redudant and useless while using \_id as a target ?

`@metadata` fields are not part of the final document that is sent to Elasticsearch, besides that, the `_id` field is set during the request not from a field in the document., so you would still need to use the `document_id` option in the output.

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [December 10, 2024, 9:42am UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712/3 "2024-12-10T09:42:36Z")

</div>

Thanks for your reply !

Would you know if there is an option in order to avoid a global elasticsearch output plugin configuration ? Not all my log sources are using/needing the fingerprint plugin.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 10, 2024, 1:42pm UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712/4 "2024-12-10T13:42:40Z")

</div>

> [@grumo35](#):
>
> Would you know if there is an option in order to avoid a global elasticsearch output plugin configuration ?

Not sure what you mean with that, can you provide more context?

Do you have different inputs and not outputs and not all of them will use a fingerprint?

I would say that the best approach is to use multiple pipelines and separate the data sources.

If you cannot or do not want to do that, then you will need to use conditionals in the output.

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [December 10, 2024, 1:45pm UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712/5 "2024-12-10T13:45:50Z")

</div>

Yess, i should work on my pipelines you just reminded me that thanks !

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 10, 2024, 1:47pm UTC](https://discuss.elastic.co/t/logstash-riddle-fingerprint-duplicates/371712/6 "2024-12-10T13:47:07Z")

</div>

Using [multiple pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) makes management easier because you do not need to rely on multiple conditionals to separate different data sources and log formats.