Logstash timestamp format

Elastic Stack Logstash
1

Hi everyone,

I have a log message as below. The date is not match any format so logstash does not parse timestamp. Anyone please help me review my logstash conf and advice please. Thanks

Log Message:
[30@00:01:33.048:] ===> INFO: Next log is "/app/svacc_pr_ctm_em/ctm_em/log/cmsg_log.CMSGATE.20220330.0", time >00:01:33.048:<

My Logstash config:

input {
beats {
port => 5044
}
}
filter
{
grok {
match => { "message" => "[%{DATA:dts}:]%{SPACE}===>%{SPACE}%{LOGLEVEL:lvl}%{GREEDYDATA:rest}" }
}
date {
match => [ "dts", "hh.mm.ss.SSS" ]
target => "@timestamp"
timezone => "UTC"
}
}
output {
stdout {
codec => rubydebug

adsasd
adsasd1408×323 10.9 KB

}
}

The out as picture above

2

Hi,

The format you've specified in the date filter, "hh.mm.ss.SSS", does not match the format of the timestamp in your log message.

Regards

3

Hi @yago82 ,

I tried date format below but it does not work. Can you give advice?

filter
{
grok {
match => { "message" => "[(?%{MONTHDAY:day}%{DATA:unknown}%{TIME}:)]%{SPACE}%{URIQUERY}%{SPACE}%{LOGLEVEL:lvl}%{GREEDYDATA:rest}" }
}
date {
match => [ "dts", "ddHH:mm:ss.SSS" ]
remove_field => [ "unknown" ]
target => "@timestamp"
timezone => "UTC"
}

4

The value of your date field dts is something like 30@00:01:33.048:, none of the date patterns you shared will match this.

You need to build a pattern that will match this string, but you need to provide more context about how this is created.

Is the 30@ part constant or it can change? If this is constant the 30@ part needs to be in your date pattern.

The following filter would parse it:

    date {
        match => ["message", "'30@'HH:mm:ss.SSS"]
    }

Keep in mind that your date string does not have any date information, so if you use the date filter in this case all your dates will be in 2024-01-01 as the example below:

{
       "dts" => "30@00:01:33.048",
      "@version" => "1",
    "@timestamp" => 2024-01-01T03:01:33.048Z,
}
5

Thanks @leandrojmp , it works. I have another log message with multiple lines. I would like to merge all to one line message. Could you please look at my Grok and advice

Log: "[timestamp: 1621431760] abort handler of pid 1823 thread 1848977280
*** Stacks of threads *** (current thread is 1848977280)
Stack of thread=1848977280, depth=3
main
shutdownServices
EMThriftServer::stop"

My Grok: [%{DATA}:%{SPACE}%{NUMBER:dts}]%{SPACE}%{GREEDYDATA:rest}