[SOLVED] Grok or not Grok on SophosXG

tardis · August 24, 2018, 11:05am

Hi,
apologies but I'm struggling to find a good reference document with examples, and I'm basing this on test and fail from a pattern test tool.
I'm trying to parse a SophosXG log message, as this one below:
device="SFW" log_type="Firewall" src_ip=111.222.333.444

there's many fields defined on the same structure TAG=VALUE
is there a way to get all tags and associated values?
Right now I'm having to use (device=%{QS:device_01}) entries for each value, which is slow and painful.

Any help or direction to the right documentation would be appreciated.

Christian_Dahlqvist · August 24, 2018, 11:12am

Look at using the kv filter instead of grok for this type of data.

rijinmp · August 24, 2018, 11:15am

Grok Patterns

https://grokdebug.herokuapp.com/patterns#

github.com

elastic/logstash/blob/v1.4.2/patterns/grok-patterns

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b

POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}

# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})

This file has been truncated. show original

rijinmp · August 24, 2018, 11:17am

For online grok parsing

https://grokdebug.herokuapp.com/

tardis · August 26, 2018, 8:02am

Kv filter using whitespace worked like charm. Thank you.

if [type] == "sophosxg" {
kv {
whitespace => strict
}
}

system · September 23, 2018, 8:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.