Logstash fingerprint plugin with JSON data

Elastic Stack Logstash
1

Hi,

I am parsing the JSON logs using below config:

  file {
        type =>"logstash"
        path => ["/logstash/*.json"]
        sincedb_path =>"/logstash/var/json.sincedb"
        codec => json
        start_position => "beginning"
  }

Later in the pipeline, I would like to use fingerprint plugin in order to avoid duplicates.
Something like this:

filter {
  fingerprint {
    source => "message"
    target => "[@metadata][fingerprint]"
    method => "MURMUR3"
  }
}
output {
  elasticsearch {
    hosts => "example.com"
    document_id => "%{[@metadata][fingerprint]}"
  }
}

But, message field is not available since the logs are being parsed using json codec.
How can I use fingerprint plugin on entire json event (all json fields) ?

Please advise. Thanks !!

2

Instead of using the JSON codec, you could read it in as a string and then run fingerprint on the text before using a json filter to parse the data. I would also recommend using a longer hash, e.g. SHA1 instead of MURMUR in order to reduce the hash collision probability.