Logstash Duplicate message

Elastic Stack Logstash
3

For duplicate messages, try using this:

filter {

  uuid {
  target => "@uuid"
  overwrite => true
  }
  fingerprint {
    source => ["message"]
    target => "fingerprint"
    key => "78787878"
    method => "SHA1"
    concatenate_sources => true
  }
  
}
output {
  elasticsearch { 
  host => localhost 
  document_id => "%{fingerprint}"
  }
  stdout { codec => rubydebug }
}

This will create a hash of your message and then if there is an exact duplicate then it will overwrite it. (In case you don't want two logs with the same contents at different times, you can use mutate{} https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html to remove the timestamp)

As for the timestamp, try using date {} http://stackoverflow.com/questions/26035136/logstash-custom-date-log-format-match

1 Like