Required logstash patterns to extract JSON format log

Hi All,

Below is the log message which is in JSON format, would like to extract couple of keys from the below log ( say ex: x-forwarded-for, host, etc..). can any please help me to write logstash filters to extract the fields.
Please help on this.

"_id":"c92f317e-2bcb-450e-b725-ff1fabdeff53-219266","timestamp":"2020-11-16T11:00:37.473Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"c92r017e-2bgb-490e-b715-ff1fabdeff93-214864","trackingIds":["c92f027e-2bcb-490e-b725-ff8fabdwff93-219937"],"userId":"id=kckadmin,ou=user,dc=openam,dc=forgerock,dc=org","client":{​​​​​"ip":"10.124.27.291","port":34920}​​​​​,"server":{​​​​​"ip":"10.133.80.179","port":8443}​​​​​,"http":{​​​​​"request":{​​​​​"secure":true,"method":"GET","path":"https://openam.dev.pointsection.com/opensso/json/realms/root/users/amadmin","headers":{​​​​​"accept":["application/json, text/javascript, /; q=0.01"],"accept-api-version":["protocol=1.0,resource=2.0"],"content-type":["application/json"],"host":["openam.dev.pointsection.com"],"user-agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"],"x-forwarded-for":["184.143.116.132"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-requested-with":["XMLHttpRequest"]}​​​​​}​​​​​}​​​​​,"request":{​​​​​"protocol":"CREST","operation":"READ"}​​​​​,"response":{​​​​​"status":"SUCCESSFUL","statusCode":"","elapsedTime":2,"elapsedTimeUnits":"MILLISECONDS","detail":{​​​​​"objectId":"id=kckadmin,ou=user,dc=openam,dc=forgerock,dc=org","revision":"-1"}​​​​​}​​​​​,"realm":"/","component":"Users"}​​​​​

Thanks
Nick

If your message was valid JSON (which as shown, it is not) then you could do something like

    json { source => "message" remove_field => [ "message" ] }
    ruby {
        code => '
            h = event.get("[http][request][headers]")
            if h
                h.each { |k, v|
                    event.set("[headers][#{k}]", v[0])
                }
            end
        '
    }

to get

      "headers" => {
    "accept-api-version" => "protocol=1.0,resource=2.0",
          "content-type" => "application/json",
      "x-forwarded-port" => "443",
      "x-requested-with" => "XMLHttpRequest",
                  "host" => "openam.dev.pointsection.com",
       "x-forwarded-for" => "184.143.116.132",
                "accept" => "application/json, text/javascript, /; q=0.01",
     "x-forwarded-proto" => "https",
            "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
}

Thanks Badger, it worked:)

Also, can we apply the same json to the below format. I don't think it work for below log. I want to the below log data as separate columns.

Please help me out. Thanks in advance.

Original log message:

[08/Dec/2020:01:36:56 +0000] category=CORE severity=ERROR msgID=200 msg=Entry "uid=test123,ou=people,dc=taphill,dc=com" contains a value "USA" for attribute c that is invalid according to the syntax for that attribute: The provided value "USA" is not a valid country string because the length is not exactly two characters.

date: [08/Dec/2020:01:36:56 +0000] or 08/Dec/2020:01:36:56 +0000
category= CORE
severity=Error
msgID=200
msg=Entry "uid=test123,ou=people,dc=taphill,dc=com" contains a value "USA" for attribute c that is invalid according to the syntax for that attribute: The provided value "USA" is not a valid country string because the length is not exactly two characters

Thanks
Nick

It would be specific to that format, but you could do it using

    kv { include_keys => [ "category", "severity", "msgID" ] }
    grok { match => { "message" => "msg=%{GREEDYDATA:msg}" } }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.