GROK pattern

Hello,

I have trouble with writing GROK pattern for system logs. My goal is to parse logs in form "systemctl -o verbose" which looks like this

Wed 2022-10-12 09:08:42.759756
    _TRANSPORT=kernel
    SYSLOG_IDENTIFIER=kernel
    _HOSTNAME=amvscore1
    PRIORITY=6
    MESSAGE=2022-10-12 09:08:42       INFO    instance/beat.go:498    filebeat stopped

So far I was able to match only date with %{TIMESTAMP_ISO8601:Time}, but I don't know how to match other fields. Each field is in new line.

Can anyone help me out please ?
Thank you.

Can you put few more, 3-4 samples?

Here's the start of a pattern. I didn't do the whole thing but it should give u the idea of how to continue.

%{DAY} %{TIMESTAMP_ISO8601:date}\n%{SPACE}_TRANSPORT=%{NOTSPACE:transport}\n%{SPACE}SYSLOG_IDENTIFIER=%{NOTSPACE:syslog_id}\n%{SPACE}_HOSTNAME=%{NOTSPACE:hostname}

Hello @Rios,

Every log is exactly the same. I mean the exact same structure because it is log from journalctl.
This is the only way how to parse fields from all systemd logs.

OK. Have you tested legoguy1000 grok pattern?

@Rios

Yes I have, that one didn't work for me, but It gave me enough to continue.

%{DAY} %{TIMESTAMP_ISO8601:date}|(\n%{SPACE})|_TRANSPORT=%{NOTSPACE:transport}|(\n%{SPACE})|SYSLOG_IDENTIFIER=%{NOTSPACE:syslog_id}|(\n%{SPACE})|_HOSTNAME=%{NOTSPACE:hostname}|
(\n%{SPACE})|PRIORITY=%{NOTSPACE:level}

Although, I can't parse the content of "MESSAGE" but I will figure it out :slight_smile:

What I don't understand is why I am able to match almost everything in pattern I posted in https://grokdebugger.com/ but I can match only timestamp in kibana dev tools. I tried legoguy1000's pattern in kibana as well, but that didnt work at all

Can you dump from the debugger, what you receive in an original message?

I am just trying to parse this log in online grok debugger or kibana grok debugger. I didn't put this pattern in logstash yet

but not working in kibana ... perhaps grok debugger in kibana thinks each line is new log ?

Run your LS conf read what you receive in Ruby debug or save in file without any filtering.

Thanks, I discovered that filebeat supports sending journald logs and I got it working :slight_smile:
Everything is working :slight_smile: we can close this topic

Thank you for all the help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.