Backfilling logs and timestamp

Elastic Stack Logstash
1

Hi everyone,

I have some issue with the "@timestamp" field when i'm importing old logs. The thing I want to do is to replace the value of the "@timestamp" with the value of when the log was created (here i'm looking at "start_time" field).

Here's a log sample of what i'm trying to parse.
Jun 1 00:00:00 10.1.1.1 SOMEDATA start_time="2016-06-06 00:00:00"

Here's my configuration for parsing the logs

filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} SOMEDATA start_time=%{QUOTEDSTRING:start_time} " }
remove_field => [ "message" ]
}
mutate {
gsub => [ "start_time", """, "" ]
}
date {
match => [ "start_time", "YYYY-MM-dd HH:mm:ss" ]
remove_field => [ "start_time" ]
}
}

I'm not really sure about those two points :

  • Can I replace the @timestamp with my "start_time" even if they don't have the same format ?
  • Can my problem be related to the type of the "start_time" (I'm using the QUOTEDSTRING) ?

Thanks in advance.

2

Can I replace the @timestamp with my "start_time" even if they don't have the same format ?

Yes. The whole point of the date filter is to parse timestamps in various formats into one canonical representation.

Can my problem be related to the type of the "start_time" (I'm using the QUOTEDSTRING) ?

Yes. IIRC the QUOTEDSTRING pattern includes the quote in the capture, i.e. the start_time field will contain "2016-06-06 00:00:00" instead of just 2016-06-06 00:00:00. You can take that into account in the date filter but because of the configuration language's support for quoting and escapes you might want to do it by other means, like using something other than QUOTEDSTRING.