Extract date from message [SOLVED]

[javier-lopez (Javier López)]

Hello forum,

I'm new to logstash and having issues trying to extract a log substring as timestamp.

The following set May 22 as the @timestamp value:

echo "May 22 12:42:16" | ./logstash-5.4.0/bin/logstash -e "input { stdin {} } filter { date { match => [ \"message\", \"MMM dd HH:mm:ss\", \"MMM dd HH:mm:ss'-Thread'\"] } }"

{
    "@timestamp" => 2017-05-22T12:42:16.000Z,
      "@version" => "1",
          "host" => "ubuntu-16",
       "message" => "May 22 12:42:16"
}

However the following fails and sets the timestamp value to the current date:

echo "May 22 12:42:16-Thread 24-Project Project/initial: Component PaymentsOnlineProcess: [LG_NRM]__======>DEBUG [Update QUERY]oracle.jdbc.driver.T4CPreparedStatement@10c44d3" | ./logstash-5.4.0/bin/logstash -e "input { stdin {} } filter { date { match => [ \"message\", \"MMM dd HH:mm:ss\", \"MMM dd HH:mm:ss'-Thread'\"] } }"

{
    "@timestamp" => 2017-05-26T20:20:48.294Z,
      "@version" => "1",
          "host" => "ubuntu-16",
       "message" => "May 22 12:42:16-Thread 24-Project Project/initial: Component PaymentsOnlineProcess: [LG_NRM]__======>DEBUG [Update QUERY]oracle.jdbc.driver.T4CPreparedStatement@10c44d3",
          "tags" => [
        [0] "_dateparsefailure"
    ]
}

Do I need to do something else in order to extract the timestamp when it's surrounded by context?

[javier-lopez (Javier López)]

Testing a little bit more found the answer, probably it's well documented somewhere else but personally couldn't find it.

echo "May 22 12:42:16-Thread 24-Project Project/initial: Component PaymentsOnlineProcess: [LG_NRM]__======>DEBUG [Update QUERY]oracle.jdbc.driver.T4CPreparedStatement@10c44d3" | ./logstash-5.4.0/bin/logstash -e "input { stdin {} } filter { grok { match => [\"message\", \"%{GREEDYDATA:customTimeStamp}-Thread%{GREEDYDATA}\"] } date { match => [ \"customTimeStamp\", \"MMM dd HH:mm:ss\"] } }"

{
         "@timestamp" => 2017-05-22T12:42:16.000Z,
           "@version" => "1",
               "host" => "ubuntu-16",
            "message" => "May 22 12:42:16-Thread 24-Project Project/initial: Component PaymentsOnlineProcess: [LG_NRM]__======>DEBUG [Update QUERY]oracle.jdbc.driver.T4CPreparedStatement@10c44d3",
    "customTimeStamp" => "May 22 12:42:16"
}

Here, a customTimeStamp field is created from the original log and use it later on the date module to update the @timestamp field.