Timestamp pattern not working for logstash filtering

Elastic Stack Logstash
1

I have 3 lines of logs with different structure and i am constructing a grok pattern to filter the logs. But the pattern I have isn't working. Even when I use just the timestamp pattern, it doesn't match.

[2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..
[2022-10-04 21:45:22 +0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)

I am expecting:

timestamp:
loglevel:
message:

But as I mentioned, even when I use just the timestamp pattern below to test on grokdebugger, it doesn't match for any of the logs:

This is the current pattern i have:
\[%{TIMESTAMP_ISO8601:timestamp}\]\:%{LOGLEVEL:loglevel}%{WORD: class} %{SPACE}%{GREEDYDATA:logMessage}

2

Your grok pattern has a closing ] on the timestamp But your logs do not on the first two lines.
3rd Line is completely different

Perhaps start with these 2 patterns and build from there

\[%{TIMESTAMP_ISO8601:timestamp}: %{LOGLEVEL:loglevel}/%{WORD:class}\] %{GREEDYDATA:logMessage}
\[%{TOMCAT_DATESTAMP:timetamp}\] \[%{POSINT:bytes}\] \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:logMessage}

Not sure if that 3094 is bytes etc...

Highly Recommend this tool incremental construction
https://grokconstructor.appspot.com/do/construction

{
  "class": "MainProcess",
  "loglevel": "INFO",
  "logMessage": "Events of group {task} enabled by remote",
  "timestamp": "2022-10-04 21:45:27,444"
}

{
  "class": "MainProcess",
  "loglevel": "ERROR",
  "logMessage": "consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..",
  "timestamp": "2022-10-04 21:43:06,521"
}

{
  "timetamp": "2022-10-04 21:45:22 +0000",
  "bytes": "3094",
  "loglevel": "INFO",
  "logMessage": "Listening at: http://0.0.0.0:8793 (3094)"
}

When / If you use the date filter to convert to a date field you will need to provide matching time patterns... what this does is just extract the timestamp as a string.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.