Time without a date field

Elastic Stack Logstash
#1

I have logs that only have a time field i.e. HH:mm:ss.SSS but for some reason there are no dates! I want to include today's date to this field called logstoretime. Because if I do not specify a date filter the time difference from when log gets shipped to when it is processed is to large of a gap ~10minutes. So essentially I want it to log by depending on the day it will pull that date and use my HH:mm:ss.SSS

Anyone with this experience?

No default Year/month/day for only HH:mm:ss,SSS logs
#2

You can do this with a date filter, eg;

date {
  match => [ "logstoretime", "HH:mm:ss.SSS" ]
}
#3

Yup figured an easy way out ( had this problem earlier in a post...but its time to really fix this issue)

what you can do is add a field

 add_field => {"logdate" => "%{+YYYY-MM-dd}"}

I can merge the 2 fields so now the issue is how do I would use a date filter. Because now my issue is when I merge the two fields it becomes an array. What I know from date filter it does not play nicely with arrays. So need to cleanly add this date without turning into a messy array or try and append this field to logstoretime.

Thanks.
M

Some extra details on what it will look like

     "logdate" => [
    [0] "2015-08-12",
    [1] "07:46:18,960"
]

  "@timestamp" => "1970-01-01T07:46:18.960Z",
#4

If anyone hits this thread, this is what I'm currently using

...
mutate {

this will only work on logs created the same day as read

add_field => {"logdate" => "%{+YYYY-MM-dd}"}

merge with existing time field

add_field => {"App_timestamp2" => "%{logdate}_%{App_timestamp}"}
}
date {
match => ["App_timestamp2", "YYYY-MM-dd_HH:mm:ss.SSS"]
timezone => "America/Chicago"

remove temporary date_time field

remove_field => ["logdate","App_timestamp2"]
}
...

2 Likes
#5