GROK pattern

Elastic Observability Logs
1

Hello,

I have trouble with writing GROK pattern for system logs. My goal is to parse logs in form "systemctl -o verbose" which looks like this

Wed 2022-10-12 09:08:42.759756
    _TRANSPORT=kernel
    SYSLOG_IDENTIFIER=kernel
    _HOSTNAME=amvscore1
    PRIORITY=6
    MESSAGE=2022-10-12 09:08:42       INFO    instance/beat.go:498    filebeat stopped

So far I was able to match only date with %{TIMESTAMP_ISO8601:Time}, but I don't know how to match other fields. Each field is in new line.

Can anyone help me out please ?
Thank you.

2

Can you put few more, 3-4 samples?

3

Here's the start of a pattern. I didn't do the whole thing but it should give u the idea of how to continue.

%{DAY} %{TIMESTAMP_ISO8601:date}\n%{SPACE}_TRANSPORT=%{NOTSPACE:transport}\n%{SPACE}SYSLOG_IDENTIFIER=%{NOTSPACE:syslog_id}\n%{SPACE}_HOSTNAME=%{NOTSPACE:hostname}
4

Hello @Rios,

Every log is exactly the same. I mean the exact same structure because it is log from journalctl.
This is the only way how to parse fields from all systemd logs.

5

OK. Have you tested legoguy1000 grok pattern?

6

@Rios

Yes I have, that one didn't work for me, but It gave me enough to continue.

%{DAY} %{TIMESTAMP_ISO8601:date}|(\n%{SPACE})|_TRANSPORT=%{NOTSPACE:transport}|(\n%{SPACE})|SYSLOG_IDENTIFIER=%{NOTSPACE:syslog_id}|(\n%{SPACE})|_HOSTNAME=%{NOTSPACE:hostname}|
(\n%{SPACE})|PRIORITY=%{NOTSPACE:level}

Although, I can't parse the content of "MESSAGE" but I will figure it out :slight_smile:

What I don't understand is why I am able to match almost everything in pattern I posted in https://grokdebugger.com/ but I can match only timestamp in kibana dev tools. I tried legoguy1000's pattern in kibana as well, but that didnt work at all

7

Can you dump from the debugger, what you receive in an original message?

8

I am just trying to parse this log in online grok debugger or kibana grok debugger. I didn't put this pattern in logstash yet

image
image1916×625 89.2 KB

but not working in kibana ... perhaps grok debugger in kibana thinks each line is new log ?

image
image800×791 34.3 KB

9

Run your LS conf read what you receive in Ruby debug or save in file without any filtering.

10

Thanks, I discovered that filebeat supports sending journald logs and I got it working :slight_smile:
Everything is working :slight_smile: we can close this topic

Thank you for all the help.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.