Help with using Grok to parse these three log formats

[roman-tasi]

Hi I am experienced with Dissect but not Grok. I think I need to use Grok here because the log format varies between the logs, so dissect will only work on one format, not all three:

Case 1:

2023-03-31 00:01:24,366 INFO [qtp509886383-311239:https://128.171.103.69:443/Microsoft-Server-ActiveSync?User=jonathan&DeviceId=LMJ0U2G6OL6HN3T5A5N89GC5QG&DeviceType=iPhone&Cmd=Sync] [ip=47.173.247.240;] security - cmd=Auth; account=jonathan@whtasi.org; protocol=zsync;

Case 2:

2023-03-31 00:00:13,319 INFO [qtp509886383-311183:http://127.0.0.1:88/service/soap/AuthRequest] [name=beau@whtasi.org;oip=66.95.159.176;ua=zclient/8.5.0_GA_3042;] security - cmd=Auth; account=beau@whtasi.org; protocol=soap;

Case 3:

2023-03-31 19:20:53,401 INFO [ImapSSLServer-1368] [ip=167.170.57.114;] security - cmd=Auth; account=patricia@uhtasi.org; protocol=imap;

If you can give me a pattern that would work, and explain it, thank you!

Oh I forgot that I want to parse out the ip (for the Case 1 and Case 3 logs), DeviceType (for the Case 1 log), oip (for the Case 2 log), account (all Cases), and protocol (all Cases) fields.

[Badger]

@roman-tasi

If you can give me a pattern that would work

Don't try to match the entire line with a pattern. Just pull out the fields you need

    grok {
        break_on_match => false
        match => {
            "message" => [
                "account=(?<account>[^;]*);",
                "DeviceType=%{WORD:deviceType}",
                "\[ip=%{IPV4:ip}",
                "oip=%{IPV4:oip}",
                "protocol=%{WORD:protocol}"
            ]
        }
    }

[roman-tasi]

Hi that solved it. But can you also help with this:

Case 1
Apr 1 15:45:09 mail postfix/smtps/smtpd[28960]: warning: unknown[113.160.218.194]: SASL LOGIN authentication failed: authentication failure

Case 2
Apr 1 15:44:51 mail postfix/smtps/smtpd[28960]: warning: cpe-172-91-158-20.socal.res.rr.com[172.91.158.20]: SASL LOGIN authentication failed: authentication failure

Case 3
Apr 1 15:43:24 mail postfix/smtps/smtpd[28960]: warning: 94-26-146-179.arbital.ru[94.26.146.179]: SASL LOGIN authentication failed: authentication failure

I want to pull out the ip field as well as the entire "SASL LOGIN authentication failed: authentication failure"

Thanks!

[Sunile_Manjee]

If you are interested in Elastic providing you a grok pattern based on your logs, take a look at this. Using Elastic ML to generate Grok patterns for log sources - YouTube

[Badger]

If you want to unconditionally pull out an IP address if present then use "%{IPV4:ip}",.

[roman-tasi]

But then how do I pull out the "SASL LOGIN authentication failed: authentication failure" ??

[Badger]

I don't know. What about those lines tells you that you want to capture everything after the [172.91.158.20]: ?

[roman-tasi]

Would it be this? "]: =(?<error>[^:]*):"

This is basically only getting the SASL LOGIN authentication failed and cutting out the : authentication failure

I'm searching for the ]: after the IP which leads into the SASL LOGIN authentication failed

[Badger]

If you want the whole of the rest of the line then use ]: %{GREEDYDATA:error}, but that is going to match any line that contains ]: .