Grok Timestamp

[Alex_Leong]

I have sample log as follows, and trying to grok it so that the timestamp is in the format YYYY-MM-DD HH:MI:SS.ssssss.   Hence the timestamp "0204 14:29:42.248281" becomes "2021-02-04 14:29:42.24828"

Sample log :

E0204 14:29:42.248281       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.3/tools/cache/reflector.go:156: Failed to watch *v1.Role: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kube-system:replicator" cannot list resource "roles" in API group "rbac.authorization.k8s.io" at the cluster scope  

custom grok pattern :
K8CUST_DATE %{MONTHNUM}%{MONTHDAY} %{TIME}

grok :
%{K8CUST_DATE:log_timestamp}%{GREEDYDATA}

output :
{
  "log_timestamp": "0204 14:29:42.248281"
}

This is as far as I got, need help with further processing to get the desired format.  I am using the grok debugger.


Thank you

Alex

[Badger]

Your timestamp does not include a year, so you are going to have to add one. I would use a date filter to do that. There is an issue on github here that discusses some of the complexities of that. For example if you are in the first minute of a new year and an event has a timestamp from December 31st it is probably from the previous year. But basically it is making an educated guess about which year should be added and sometimes it will get it wrong.

A date filter only keeps millisecond accuracy, so if you want more than that you will have to merge the time from the original field. The filter below parses the date, to get the year added, then converts the result back to a string. It then extracts the first 10 characters of it (2021-02-04) and combines it with everything except the first 5 characters of log_timestamp (removing the 0204 )

    date { match => [ "log_timestamp", "MMdd HH:mm:ss.SSSSSS" ] target => "[@metadata][timestamp]" }
    mutate { convert => { "[@metadata][timestamp]" => "string" } }
    mutate {
        gsub => [
            "[@metadata][timestamp]", "(.{10}).*", "\1",
            "log_timestamp", ".{5}(.*)", "\1"
        ]
    }
    mutate { replace => { "log_timestamp" => "%{[@metadata][timestamp]} %{log_timestamp}" } }

If you really do want 5 digits of sub-second precision instead of 6 then

    mutate { gsub => [ "log_timestamp", "\d$", "" ] }