Parse string to date and have difference between 2 dates


(Pierre “What's Up” Scadra) #1

Hi !
I have a string with this format :

Fri Feb 16 14:30:09 CET 2018

I init a field like this :

mutate {
add_field => { "dateLog" => "%{message}" }
}
mutate {
gsub => [ "dateLog", "INFOS:\s", "" ]
}
mutate {
gsub => [ "dateLog", "\s([0-9.])\s([A-Z])\s([^ ])(\s)([^ ])\s([^ ])\s([^ ]\s[A-Za-z]\s[0-9]\s[0-9:]\s[A-Z]\s[0-9])\s([0-9]{3})\s([0-9.-])\s([0-9.-])\s[A-Za-z0-9-]\s[^ ]\s[^ ]\s[^ ]\s", "" ]
}

The string format is ok, no problem with this but I need to convert it to date, and have the difference with @timestamp.

How can I do ?

Do you have an exemple for me ?


(Cody Burke) #2

Have you looked into the date filter to convert the string into a date more simply than your regex?
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

From there, I think you could use the elapsed plugin to find the time difference you want. https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-elapsed.html


(Pierre “What's Up” Scadra) #3

thank you for your reply : I tried this but it does not work :
date {
match => [ "dateLog", "MMM d H:mm:ss yyyy" ]
}


#4

That does not match "MMM d H:mm:ss yyyy". There is a day of the week (EEE) and an identity timezone (ZZZ) in there.


(Robert Cowart) #5

I used this recently at a customer, and it might help point you in the right direction...

# Convert the dates to a proper timestamp.
date {
  match => [ "BEGIN", "dd/MM/YYYY HH:mm:ss.SSSSSS" ]
  timezone => "UTC"
  target => "BEGIN"
}
date {
  match => [ "LAST", "dd/MM/YYYY HH:mm:ss.SSSSSS" ]
  timezone => "UTC"
  target => "LAST"
}

# If the dates parsed properly calculate the runtime in seconds (precision is ms).
if "_dateparsefailure" not in [tags] {
  ruby {
    code => "
      event.set('[RUNTIME_SEC]', (event.get('[LAST]').to_f - event.get('[BEGIN]').to_f).round(3));
    "
  }
}

(Pierre “What's Up” Scadra) #6

So how can I do ?


(Pierre “What's Up” Scadra) #7

Thank you for your reply I will test it when I succeed to parse my string


#8

That would match if dateLog were something like "Feb 16 14:30:09 2018", but you said it is formatted as "Fri Feb 16 14:30:09 CET 2018". So you need to modify the pattern to include items to match the "Fri" and "CET". The pattern has to match every part of the date string.


(Pierre “What's Up” Scadra) #9
     "dateLog" => "Feb 16 14:30:09 2018",

I have change the format like you say, but nothing has changed .... I do not have a field date


#10

So what exactly does your configuration look like? If I run logstash with this configuration

input { stdin { } }
output { stdout { codec => rubydebug } }
filter { date { match => [ "message", "MMM d H:mm:ss yyyy" ] } }

and feed it Feb 16 14:30:09 2018, then it gets parsed...

    "@timestamp" => 2018-02-16T19:30:09.000Z,
       "message" => "Feb 16 14:30:09 2018",

(Pierre “What's Up” Scadra) #11

Yes but how can I inject :
"message" => "Feb 16 14:30:09 2018",
in date format inside ElasticSearch ?


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.