LogStash filter for matching timestamp example: [2024-01-04 23:00:00,931]

[criss79 (Criss)]

Hi guys,
I am having difficulties to match this timestamp format for a log entry that looks like this:
[timestamp] [Loglevel] message
Log entry example:
[2024-01-04 23:00:00,931] [INFO] Multi_Language.UserInfoContainer PushNotificationChannels.SpreadsUpdated with .DistributedCache.Common.PublisherInfoExtened

My Logstash filter:

filter {
    grok {
    match => { "message" => "^\[(?<timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+(\[%{WORD:loglevel}\]\s+)?%{GREEDYDATA:message}" }
    overwrite => [ "message" ]
    }
  kv {
    field_split => " "
}
  mutate {
    remove_field => ["event","log","input","ecs","version","name","@version","input","type","agent","offset","tags"]
    lowercase => [ "[host][name]" ]
}
}

For the following log entry example the filter above works but this will not help me as I need the timestamp in this format “[2024-01-04 23:00:00,931]”.

[Thu Nov 01 21:56:35 2012] [INFO] Multi_Language.UserInfoContainer PushNotificationChannels.SpreadsUpdated with .DistributedCache.Common.PublisherInfoExtened

I appreciate any ideas.

Have a nice day all.

[yago82]

Hi,

The grok pattern you're using is for a different timestamp format.

Here's a modified version of your Logstash filter that should work with your timestamp format:

filter {
  grok {
    match => { "message" => "^\[(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})\]\s+\[%{LOGLEVEL:loglevel}\]\s+%{GREEDYDATA:message}" }
    overwrite => [ "message" ]
  }
  date {
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
    target => "@timestamp"
    remove_field => [ "timestamp" ]
  }

Regards

[criss79 (Criss)]

Thank you yago82,

Your filter seems to be correct, no errors, however there are no entries in elastic db. I searched all indexes and data streams.

I found the following filter and is handling my timestamp correctly. I also changed the separators since my logs contains characters like [ ] { }.

filter {
  dissect {
    mapping => {
        "message" => "|%{time}| |%{level}| |%{logmsg}|"
      }
}
  kv {
    field_split => " "
}
  mutate {
    remove_field => ["event","log","input","ecs","version","name","@version","input","type","agent","offset","tags"]
    lowercase => [ "[host][name]" ]
}
}

Thanks again.
Criss.