Logstash 7.10 fails with date filter UNIX_MS - failed to parse date field strict_date_optional_time

[dfrewin (Derek)]

Initially logstash was running when we were on 6.8 but as soon as we upgraded to 7.9.3 it started failing.

Using salesforce logstash plugin error msg retrieves the time value successfully, but looks to fail when sending to elastic search.

failed to parse date field [1.606245613433E12] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

logstash Conf file :

filter {
  date { tag_on_failure => []
   match => ["CreatedTimeStamp__c", "UNIX_MS"] }

  mutate {
      add_field => { "logtype" => "salesforce" }
      rename => { "Message__c" => "message" }
      rename => { "Level__c" => "loglevel" }
          remove_field => [ "CreateTimeStamp__c"]
}

[Badger]

@dfrewin

date { tag_on_failure => []
   match => ["CreatedTimeStamp__c", "UNIX_MS"] }

I do not know what changed to stop it parsing the exponential format, but if you want to get it working again you can

 mutate { convert => { "CreatedTimeStamp__c" => "float" } }

so that you have

"CreatedTimeStamp__c" => 1606245613433.0

which the date filter will parse as 2020-11-24T19:20:13.433Z

[dfrewin (Derek)]

@Badger thank you for the response. I did as you said but it still was failing. I then piped the output to stdout to see what I as actually sending to elastic search. This revealed a bug in my configuration.

My remove statement had CreateTimeStamp__c instead of CreatedTimeStamp__c . Anyways you got me to the next step in my debugging. Thanks.