How should I parse date time for below string

Elastic Stack Logstash
1

In my file, the content is like this:

2016040417350700, > '/home/prbdata/esda/prod/spool.6/esda/f12masc1d0/7882201_001_' [1:2:1]

how can I parse 2016040417350700 into a field with correct date time format ?

2

I can parse it into each Year Month Date field, but not sure how to combine them into a complete timestamp field.

3

If you check out this page you will see that there are a lot of regex patterns already defined

http://grokdebug.herokuapp.com/patterns#

So try changing your pattern to this:

(?<runtime>%{YEAR}%{MONTHNUM}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}(.[0-9]))

This should give you a field called runtime that contains your timestamp value.

4

hi Mick, thanks for your reply. I tried on the debugger, the runtime still shows numbers 2016040417350700, not in the date time format. I think your method is to use the logstash pattern instead of custom pattern to parse year, month, day, hour, minute, second, but it is still unable to combine those fields automatically into the full date time format.

2016-04-06_10-08-30.jpg685×929 72.1 KB

5

Sorry I misunderstood your requirement. Once you have the parsed the runtime field from your message you can then use the date filter to set the timestamp like this

filter {
    grok {
       match => ["message", "(?<runtime>%{YEAR}%{MONTHNUM}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}(.[0-9]))"]
    }

    date {
        match => [ "runtime", "YYYYMMddHHmmssSS" ]
        target => "@timestamp"
    }
}
6

hi Mick,

I actually made it work in the end by below config but it looks so stupid and inefficient compared to your one. sorry I am new to logstash. Thanks for your help!

    grok {
            match => { "message" => [ "^# ((?<year>%{YEAR})(?<month>%{MONTHNUM})(?<day>%{MONTHDAY})(?<hour>%{HOUR})(?<min>%{MINUTE})(?<second>%{SECOND}))?(.[0-9])%{GREEDYDATA}"] }
     }
    grok {
            match => { "message" => [ "^# \d{16}, \[EVENT::METRIC\] 'STAGE':'(?<ProcessTime>%{BASE16FLOAT})'"]}
    }
    multiline {
            pattern => "#\s\D"
            what => "previous"

    }

    mutate {
            add_field => { "EventDate" => "%{hour}:%{min}:%{second} %{month} %{day} %{year}"}
    }

    date {
           match => ["EventDate", "HH:mm:ss MM dd yyyy"]
     }
    mutate {
            remove_field => [ "year","month","day","hour","min","second","EventDate"]
    }