Logstash fingerprint plugin with JSON data

Hi,

I am parsing the JSON logs using below config:

  file {
        type =>"logstash"
        path => ["/logstash/*.json"]
        sincedb_path =>"/logstash/var/json.sincedb"
        codec => json
        start_position => "beginning"
  }

Later in the pipeline, I would like to use fingerprint plugin in order to avoid duplicates.
Something like this:

filter {
  fingerprint {
    source => "message"
    target => "[@metadata][fingerprint]"
    method => "MURMUR3"
  }
}
output {
  elasticsearch {
    hosts => "example.com"
    document_id => "%{[@metadata][fingerprint]}"
  }
}

But, message field is not available since the logs are being parsed using json codec.
How can I use fingerprint plugin on entire json event (all json fields) ?

Please advise. Thanks !!

Instead of using the JSON codec, you could read it in as a string and then run fingerprint on the text before using a json filter to parse the data. I would also recommend using a longer hash, e.g. SHA1 instead of MURMUR in order to reduce the hash collision probability.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.