Logstash riddle fingerprint duplicates

Elastic Stack Logstash
1

Hi there,

Using the logstash fingerprint plugin i came across an issue where it seems i cannot define the _id as a target from the plugin using the following :

fingerprint {
    source => ["[host][name]", "[record][id]"]  
    target => "[@metadata][_id]"            
    method => "SHA256"                                        
    concatenate_sources => true             
  }

The configuration above strangely does not work as intented.

Do i need to document_id => "%{[@metadata][_id]}" in my elastic outptut ?

Why does it feels redudant and useless while using _id as a target ?

2

If you want to use custom ids, then yes, you need to use the document_id in your Elasticsearch output.

@metadata fields are not part of the final document that is sent to Elasticsearch, besides that, the _id field is set during the request not from a field in the document., so you would still need to use the document_id option in the output.

1 Like
3

Thanks for your reply !

Would you know if there is an option in order to avoid a global elasticsearch output plugin configuration ? Not all my log sources are using/needing the fingerprint plugin.

4

Not sure what you mean with that, can you provide more context?

Do you have different inputs and not outputs and not all of them will use a fingerprint?

I would say that the best approach is to use multiple pipelines and separate the data sources.

If you cannot or do not want to do that, then you will need to use conditionals in the output.

1 Like
5

Yess, i should work on my pipelines you just reminded me that thanks !

6

Using multiple pipelines makes management easier because you do not need to rely on multiple conditionals to separate different data sources and log formats.