Logstash config with Grok

Elastic Stack Logstash
1

Hi Everyone,

I am trying to config logstash parse the log with 3 field : date & time (dts), Log level (lvl) and rest message.
But the out put always show timestamp is not correct, timestamp i want to show is the time in the log error but it show the time of local machine. Please give me advice. Below are logstash conf anf the output FYI

-> The error sample is: 2022-03-15 12:42:28.5801 INFO 1 Com.Bmc.Ctmem.EmCSInfrustructure.EMFoundations.GetEMLogLocation C:\Program Files\BMC Software\Control-M EM\CBMTCTCAPR01_0\log\

-> logstash conf:
input {
beats {
port => 5044
}
}

filter
{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:dts}%{SPACE}%{LOGLEVEL:lvl}%{GREEDYDATA:rest}" }}
date { match => ["dts" , "yyyy-MM-dd HH:mm:ss,SSSS"]
}
}
output {
stdout {
codec => rubydebug
}
}
-> tghe output as screenshot

Screenshot 2024-05-11 135833
Screenshot 2024-05-11 1358331563×669 17.9 KB

Thanks all

2

Hi @iceman0410,

Below is the grok pattern which worked from me -

Screenshot 2024-05-11 at 6.57.36 PM
Screenshot 2024-05-11 at 6.57.36 PM3436×1014 302 KB

Logtstash default adds @timestamp field with local timestamp. You can overwrite it using date filter like below.

input {
  # Your input configuration here
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:level} %{NUMBER:info_number} %{GREEDYDATA:class} %{GREEDYDATA:file_path}" }
  }
  date {
    match => [ "log_timestamp", "yyyy-MM-dd HH:mm:ss.SSSSSSSSS" ]
    target => "@timestamp"
  }
}

output {
  # Your output configuration here
}
3

Your logtimestamp field has ss.SSSS, not ss,SSSS

Please do not post pictures of text, some folks cannot see them, they are not searchable, and we cannot copy/paste them to try to reproduce the issue.

1 Like
4

Thanks @ashishtiwari1993 , it is working now

5

Thanks @Badger ,
i added one more grok filter for another parser filter. Could you help me check logstash conf below?

Logstash conf :

input {
beats {
port => 5044
}
}

filter
{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:dts}%{SPACE}%{LOGLEVEL:lvl}%{GREEDYDATA:rest}" }}
date { match => [ "dts" , "yyyy-MM-dd HH:mm:ss.SSSS" ]
target => "@timestamp"
}
grok { match => { "message" => "(?%{DATE_EU}%{SPACE}%{TIME})%{GREEDYDATA:rest}"}}
date { match => [ "dts" , "dd/MM/yyyy HH:mm:ss.SSS" ]
tagret => "@timestamp"
}
}
output {
stdout {
codec => rubydebug
}
}

The error sample:
30/03/2022 07:50:44.590 [0] emthriftserviceclientfactory EMTHttpClientPoolInitializer::clearPool clearPool called.

Thanks everyone

6

You could try

    grok { match => { "message" => [
        "%{TIMESTAMP_ISO8601:dts} %{WORD:level} %{NUMBER:info_number} %{JAVACLASS:class} %{GREEDYDATA:file_path}",
        "(?<dts>%{DATE_EU}%{SPACE}%{TIME})%{GREEDYDATA:rest}"
        ]
    } }
    date {
        match => [ "dts", "yyyy-MM-dd HH:mm:ss.SSSSSSSSS", "dd/MM/YYYY HH:mm:ss.SSS" ]
     }
1 Like
7

Thanks @Badger , it is working

8

Hi @Badger ,
i have another log as below

0711 06:40:23.726:pid=119813:ERR:../../proto/ssl/openssl_tls.c:359:SSLsess=2472cd0:SSL_connect:SSL_ERROR_SYSCALL: write error system error 104
My grok:
%{MONTHNUM:month}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{NOTSPACE}%{LOGLEVEL:LVL}%{GREEDYDATA:REST}

output can not get pid. Could you please give advice. Thanks
[
{
"month": 7,
"day": 11,
"time": "06:40:23.726",
"LVL": "ERROR",
"REST": "_SYSCALL: write error system error 104"
}
]

9

This is throwing away the pid. Try retaining that field by naming it, then refine the pattern to extract the pid.

1 Like
10

Thanks @Badger . it is working with grok below:
?%{MONTHNUM:month}%{MONTHDAY:day}%{SPACE}%{TIME:time})%{DATA:PID}%{SPACE}%{LOGLEVEL:LVL}%{GREEDYDATA:REST}

Could you advice for another sample below. it's contained AM/PM format:
"Jul 11, 2022 6:44:49 AM org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi getDefaultTrustStore INFO: Initializing with trust store at path: /app/ctrlm-agent/ctm/cm/AFT/JRE_11/lib/security/cacerts"

Thanks

11

Define a custom pattern for the grok and then parse it in the date filter using MMM dd, yyyy hh:mm:ss a. (Note hh rather than HH since this is hour of the half-day.)

12

Hi @Badger ,

I tried:
(?%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME} (AM|PM)) %{GREEDYDATA:rest} . Can get the correct date output. Can you advice how i can split Loglevel INFO and "org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi getDefaultTrustStore" -> i don't know how can define this one

Thanks

13

It will be like one of your earlier examples, JAVACLASS will match the class name and LOGLEVEL will match the log level.

14

Hi @Badger ,
Can you check my grok as below, i tried to add JAVACLASS but it does not work

ddddddd
ddddddd1891×351 42.3 KB

Thanks

15

That grok does not use JAVACLASS, also, the alternation for AM|PM is wrong, it needs to be surrounded by parentheses.

18

Thanks @Badger . I have another log message with multi lines need to parse. Can you have a look. My Grok can not get the rest of lines

12
121667×821 103 KB

My Grok: %{LOGLEVEL:lvl}%{SPACE}%{TIMESTAMP_ISO8601:dts}%{SPACE}%{SPACE}%{GREEDYDATA:rest}

Thanks