Required logstash patterns to extract JSON format log

[nikhilesh]

Hi All,

Below is the log message which is in JSON format, would like to extract couple of keys from the below log ( say ex: x-forwarded-for, host, etc..). can any please help me to write logstash filters to extract the fields.
Please help on this.

"_id":"c92f317e-2bcb-450e-b725-ff1fabdeff53-219266","timestamp":"2020-11-16T11:00:37.473Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"c92r017e-2bgb-490e-b715-ff1fabdeff93-214864","trackingIds":["c92f027e-2bcb-490e-b725-ff8fabdwff93-219937"],"userId":"id=kckadmin,ou=user,dc=openam,dc=forgerock,dc=org","client":{​​​​​"ip":"10.124.27.291","port":34920}​​​​​,"server":{​​​​​"ip":"10.133.80.179","port":8443}​​​​​,"http":{​​​​​"request":{​​​​​"secure":true,"method":"GET","path":"https://openam.dev.pointsection.com/opensso/json/realms/root/users/amadmin","headers":{​​​​​"accept":["application/json, text/javascript, /; q=0.01"],"accept-api-version":["protocol=1.0,resource=2.0"],"content-type":["application/json"],"host":["openam.dev.pointsection.com"],"user-agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"],"x-forwarded-for":["184.143.116.132"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-requested-with":["XMLHttpRequest"]}​​​​​}​​​​​}​​​​​,"request":{​​​​​"protocol":"CREST","operation":"READ"}​​​​​,"response":{​​​​​"status":"SUCCESSFUL","statusCode":"","elapsedTime":2,"elapsedTimeUnits":"MILLISECONDS","detail":{​​​​​"objectId":"id=kckadmin,ou=user,dc=openam,dc=forgerock,dc=org","revision":"-1"}​​​​​}​​​​​,"realm":"/","component":"Users"}​​​​​

Thanks
Nick

[Badger]

If your message was valid JSON (which as shown, it is not) then you could do something like

    json { source => "message" remove_field => [ "message" ] }
    ruby {
        code => '
            h = event.get("[http][request][headers]")
            if h
                h.each { |k, v|
                    event.set("[headers][#{k}]", v[0])
                }
            end
        '
    }

to get

      "headers" => {
    "accept-api-version" => "protocol=1.0,resource=2.0",
          "content-type" => "application/json",
      "x-forwarded-port" => "443",
      "x-requested-with" => "XMLHttpRequest",
                  "host" => "openam.dev.pointsection.com",
       "x-forwarded-for" => "184.143.116.132",
                "accept" => "application/json, text/javascript, /; q=0.01",
     "x-forwarded-proto" => "https",
            "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
}

[nikhilesh]

Thanks Badger, it worked:)

Also, can we apply the same json to the below format. I don't think it work for below log. I want to the below log data as separate columns.

Please help me out. Thanks in advance.

Original log message:

[08/Dec/2020:01:36:56 +0000] category=CORE severity=ERROR msgID=200 msg=Entry "uid=test123,ou=people,dc=taphill,dc=com" contains a value "USA" for attribute c that is invalid according to the syntax for that attribute: The provided value "USA" is not a valid country string because the length is not exactly two characters.

date: [08/Dec/2020:01:36:56 +0000] or 08/Dec/2020:01:36:56 +0000
category= CORE
severity=Error
msgID=200
msg=Entry "uid=test123,ou=people,dc=taphill,dc=com" contains a value "USA" for attribute c that is invalid according to the syntax for that attribute: The provided value "USA" is not a valid country string because the length is not exactly two characters

Thanks
Nick

[Badger]

It would be specific to that format, but you could do it using

    kv { include_keys => [ "category", "severity", "msgID" ] }
    grok { match => { "message" => "msg=%{GREEDYDATA:msg}" } }