How to parse two timestamp fields from one single log line message

Elastic Stack Logstash
1

Hello All,
I have log lines coming from K8's to logstash and they have two timestamps back to back. How can I parse those fields in let's say @timestamp field and timestamp2 field?

My log lines look like:

2021-10-08 18:39:09.866 EDT 2021-10-08 18:39:09.866 [http-nio-8443] [] Filter - CorrelationFilter Start

my logstash filter looks like:

filter {
                    multiline {
                        pattern => "^\[0-9]{4}-[0-9]{2}"
                        what => "previous"
                        negate=> true
                    }
                    grok {###Not sure how to write the grok for the second timestamp
			            match => ["message", "%{DATESTAMP:timestamp}"]
                    }
                    date {
                        match => ["timestamp", "yyyy-MM-dd H:m:s.SSS"]
                        target => ["@timestamp"]
                        remove_field => ["timestamp"]
                    }
                    mutate {
                        add_field => { "type" => "K8logs" }
                           }
            }

I am having difficulties in writing the grok for the second timestamp filter.
Any help is appreciated.

Thank you.

2

I would do that using dissect

dissect { mapping => { "message" => "%{ts1} %{+ts1} %{+ts1} %{ts2} %{+ts2}" } }
mutate { gsub => [ "ts1", "EDT", "EST5EDT" ] }
date { match => [ "ts1", "YYYY-MM-dd HH:mm:ss.SSS ZZZ" ] target => "ts1" }
date { match => [ "ts2", "YYYY-MM-dd HH:mm:ss.SSS" ] target => "ts2" }

Joda does not support the ambiguous EDT timezone name, so you will need to gsub it to a non-ambiguous name.

3

Thank you Badger.
But then how will I grok filter should be?
Something like this:

grok {
	   match => ["message", "\s\[%{DATA:threadid}\]\s\[\]\s%{WORD:filter}\s-\s%{GREEDYDATA:details}"]
      }

Is this correct? I mean I don't need to specify anything for the two timestamps in grok right?

4

I am suggesting you use dissect and not use grok.

5

Ok so something like this?

dissect { mapping => { "message" => "%{ts1} %{+ts1} %{+ts1} %{ts2} %{+ts2} [%{threadid}] [] %{filter} - %{details}" } }
mutate { gsub => [ "ts1", "EDT", "EST5EDT" ] }
date { match => [ "ts1", "YYYY-MM-dd HH:mm:ss.SSS ZZZ" ] target => "ts1" }
date { match => [ "ts2", "YYYY-MM-dd HH:mm:ss.SSS" ] target => "ts2" }
6

That looks reasonable.

7

Thank you Badger, the steps you mentioned worked.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.