Grok filter extracting fields from message

[heinrich (Hein)]

Hi guys,

I've never used grok patterns before, after hours of looking into this i'm making very slow progress.

Here's a sample of what i have.

2020-06-23 15:30:02.568 [https-jsse-nio-0.0.0.0-8443-exec-58] INFO c.c.d.c.s.a.UserAuthenticationListener - User logged in: AuthenticationEvent{userName='username', sessionId='-303021888', timeout=1800000, licenseType=CONSUMER, productPermissions='bsg,rdm,dsm,catalog,helpdesk,policymanager,datadictionary,admin', action=LOGIN, remoteHost='10.251.35.112', userAgent='Chrome', userAgentVersion='58.0.3029.110', failureReason=null, timestamp=1592919002568}

I only want to extract the below information

UserAuthenticationListener - User logged in: AuthenticationEvent{userName='username', sessionId='-303021888', timeout=1800000, licenseType=CONSUMER, productPermissions='bsg,rdm,dsm,catalog,helpdesk,policymanager,datadictionary,admin', action=LOGIN, remoteHost='10.251.35.112', userAgent='Chrome', userAgentVersion='58.0.3029.110', failureReason=null, timestamp=1592919002568}

Is there anyone that could possibly be so kind to help me with a filter for this, please?

[Badger]

I would do that using dissect rather than grok

dissect { mapping => { "message" => "%{} %{} [%{}] %{} %{someField}" } }

[heinrich (Hein)]

Hi Badger,

Am i using it exactly as you provided? Apologies man i'm extremely new to this.

[Badger]

Assuming you do not want to extract the date, thread name, and log level then yes, use it exactly as I wrote it. If you want to extract those too then you would use

dissect { mapping => { "message" => "%{ts} %{+ts} [%{thread}] %{loglevel} %{someField}" } }

[heinrich (Hein)]

Hey Badger, here's what i have.

The line you provided, is that supposed to split the fields like loglevel highlighted?

[Badger]

Yes. If you want to split out fields like action, remoteHost, etc., then use a kv filter.

[heinrich (Hein)]

Is it by anyway possible you could give me one example, like action=login and userName=username

I'm trying to understand how it's formulated, please man i'm desperate to get this working

[Badger]

    dissect { mapping => { "message" => "%{ts} %{+ts} [%{thread}] %{loglevel} %{someField}{%{[@metadata][kvData]}}%{}" } }
    kv {
        source => "[@metadata][kvData]"
        field_split_pattern => ", "
    }

I changed the dissect to pull out the data between AuthenticationEvent{ and } into a separate field. That will result in

         "someField" => "c.c.d.c.s.a.UserAuthenticationListener - User logged in: AuthenticationEvent",
          "userName" => "username",
          "loglevel" => "INFO",
            "action" => "LOGIN",
         "userAgent" => "Chrome",
  "userAgentVersion" => "58.0.3029.110",
"productPermissions" => "bsg,rdm,dsm,catalog,helpdesk,policymanager,datadictionary,admin",
       "licenseType" => "CONSUMER",
     "failureReason" => "null",
           "timeout" => "1800000",
        "remoteHost" => "10.251.35.112",
         "timestamp" => "1592919002568",
         "sessionId" => "-303021888",
            "thread" => "https-jsse-nio-0.0.0.0-8443-exec-58",
                "ts" => "2020-06-23 15:30:02.568"

[heinrich (Hein)]

YOU ARE A LEGEND! THANK YOU so so much