"uuid" and "fingerprint" fields

Elastic Stack Logstash
1

Hi Team,

I am using below configuration in logstash.conf for avoiding duplicate log entries:
filter{
uuid {
target => "@uuid"
overwrite => true
}
fingerprint {
source => ["message"]
target => "fingerprint"
key => "78787878"
method => "SHA1"
concatenate_sources => true
target => "[@metadata][fingerprint]"
}

}}
output {
elasticsearch {
host => localhost
document_id => "%{[@metadata][fingerprint]}"
}
stdout { codec => rubydebug }
}

Is there any issues on using this method.?

2

I don't remember if you need to add an extra option to the ES output to get upsert behavior. Apart from that possible issue this should be fine except. What's the purpose of the uuid field?

3

inorder to remove the duplication

4

But

  • UUIDs are random and therefore useless for duplicate elimination, and
  • you're not using the uuid field anywhere in the rest of your configuration.
5

so you mean "fingerprint" field is enough for removing the duplication?

6

Yes.

7

ohk....will it impact anything except duplication ?

8

Will what impact anything? The mere presence of the [@metadata][fingerprint] field?