Fingerprint unable to fingerprint @timestamp


(Charles Patton) #1

With input lines like:

{"count":10, "timestamp":"2015-09-22T00:00:00.000Z"}

I am unable to fingerprint based off of the date extracted by my date filter. For fingerprint, it's exactly the same for all of my messages when based off of @timestamp but it is in-fact unique when I use the timestamp field (or count for that matter).

I was able to confirm that the @timestamp field is being created successfully off of the date filter so hopefully I'm just missing something here syntactically?

My config:

filter {
  date {
    match => ['timestamp', 'ISO8601']
  }
# also played with checksum
#  checksum {
#    algorithm => 'md5' # works as expected
#  }
  fingerprint {
    method => 'MD5'
    key => '00000000'
    target => 'fingerprint'
    #source => [ 'timestamp' ] ## works!
    #source => [ 'count' ] ## works!
    source => [ '@timestamp' ] #does not work!
  }
}

output {
  stdout {
    codec => 'json'
  }
  elasticsearch {
    host => '127.0.0.1'
    cluster => 'logstash'
    document_id => '%{fingerprint}'
  }
}

(Charles Patton) #2

After posting this I did some more testing and found that the only instance in which fingerprint is not unique is when the timestamps differ in their milliseconds. Fingerprint bug?


(Mark Walkom) #3

You mean between timestamp and @timestamp aren't the same values?


(Charles Patton) #4

Sorry, no, I mean to say that a timestamp of 2015-09-22T00:00:00.000Z and 2015-09-22T00:00:00.001Z or 2015-09-22T00:00:00.980Z all share the same fingerprint result.


(system) #5