Dateparsefailure on time field

[ankitachow]

I have date and time as separate fields in my input file. Date as 2019-08-28 ("yyyy-MM-dd"). Time as 09:33:05.579476547 (HH:mm:ss.SSSSSSSSZ). I want to merge the date and time for every record and create a timestamp field which will be the @timestamp for the ES index to be indexed upon.

I use below mutate but getting a dateparsefailure.

  mutate {
     remove_field => [ "message" ]
     rename => { "[dest][date]" => "new_date" }
     remove_field => "[dest][date]"
     rename => { "[dest][time]" => "new_time" }
     remove_field => "[dest][time]"
     add_field => {
         "timestamp" => "%{new_date} %{new_time}"
     }
  }
  date {
    match => ["timestamp" , "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ"]
  }

[Badger]

Z matches a timezone, which your field does not have, and it has 9 digits subsecond, and you have a space separating date and time. Try

"yyyy-MM-dd HH:mm:ss.SSSSSSSSS"

[ankitachow]

It worked.

How can I format the ts to "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ" format?

Also, I would like to keep both ts, that is @timestamp & new_timestamp which is created from file. But when I use below mutate, I can see both the fields having value from file.

  mutate {
     remove_field => [ "message" ]
     rename => { "[dest][date]" => "new_date" }
     remove_field => "[dest][date]"
     rename => { "[dest][time]" => "new_time" }
     remove_field => "[dest][time]"
     add_field => {
         "new_timestamp" => "%{new_date} %{new_time}"
     }
  }
  date {
    match => ["new_timestamp" , "yyyy-MM-dd HH:mm:ss.SSSSSSSSS"]
  }

stdout:

{
       "@timestamp" => 2019-08-28T14:33:05.638Z,
    "new_timestamp" => "2019-08-28 09:33:05.638065493",
         "new_date" => "2019-08-28",
         "new_time" => "09:33:05.638065493"
}

[Badger]

@ankitachow

How can I format the ts to "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ" format?

I do not understand the question.

Use the target option for the date filter to overwrite new_timestamp

target => "new_timestamp"