Filebeat ingest pipeline Grok pattern

Elastic Stack Beats
1

Must be something really simple, but am struggling to make it work. :confused:
The sample log line:
2020-07-29 12:17:16.948 +02:00 [80000025-0002-ff00-b63f-84710c7967bb] [Some.Text.Goes.Here.Controllers.UserController] [DBG] Starting api/me

This timezone segment is giving me a headache. I cannot match it with TIMESTAMP_ISO8601.
If we ignore the date and timezone, so far I can Grok it with this pattern:
[%{UUID:Correlation}] [%{GREEDYDATA:Source}] [%{WORD:Level}] %{GREEDYDATA:Message}

Did somebody work with this kind of Date format?

2

I figured it out. Should be no space between the datetime and timezone

2020-07-29 12:17:16.948 +02:00 => 2020-07-29 12:17:16.948+02:00

Then this would Grok it successfully:
%{TIMESTAMP_ISO8601:timestamp} [%{UUID:Correlation}] [%{GREEDYDATA:Source}] [%{WORD:Level}] %{GREEDYDATA:Message}

3

Yes, I was writting the same.

If you can not change your logs, you have to separate the date and the timezone with this following grok pattern:

%{TIMESTAMP_ISO8601:date} %{ISO8601_TIMEZONE:timezone} \[%{UUID:Correlation}\] \[%{GREEDYDATA:Source}\] \[%{WORD:Level}\] %{GREEDYDATA:Message}

Then you have to concatenate 'date' and 'timezone' and use the date filter plugin to transform your date to a valid UTC date.

4

Thanks @Rom1! That is a better answer. :wink:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.