# Required logstash patterns to extract JSON format log **URL:** \ **Category:** Logstash\ **Created:** [November 16, 2020, 11:50am UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516 "2020-11-16T11:50:20Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![nikhilesh](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nikhilesh](https://discuss.elastic.co/u/nikhilesh)\ **Post date:** [November 16, 2020, 11:50am UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516/1 "2020-11-16T11:50:20Z")
Hi All, Below is the log message which is in JSON format, would like to extract couple of keys from the below log ( say ex: x-forwarded-for, host, etc..). can any please help me to write logstash filters to extract the fields. Please help on this. "\_id":"c92f317e-2bcb-450e-b725-ff1fabdeff53-219266","timestamp":"2020-11-16T11:00:37.473Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"c92r017e-2bgb-490e-b715-ff1fabdeff93-214864","trackingIds":["c92f027e-2bcb-490e-b725-ff8fabdwff93-219937"],"userId":"id=kckadmin,ou=user,dc=openam,dc=forgerock,dc=org","client":{​​​​​"ip":"10.124.27.291","port":34920}​​​​​,"server":{​​​​​"ip":"10.133.80.179","port":8443}​​​​​,"http":{​​​​​"request":{​​​​​"secure":true,"method":"GET","path":"[https://openam.dev.pointsection.com/opensso/json/realms/root/users/amadmin","headers":{​​​​​"accept":["application/json](https://openam.dev.pointsection.com/opensso/json/realms/root/users/amadmin%22,%22headers%22:%7B%E2%80%8B%E2%80%8B%E2%80%8B%E2%80%8B%E2%80%8B%22accept%22:%5B%22application/json), text/javascript, _/_; q=0.01"],"accept-api-version":["protocol=1.0,resource=2.0"],"content-type":["application/json"],"host":["[openam.dev.pointsection.com](http://openam.dev.pointsection.com)"],"user-agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"],"x-forwarded-for":["184.143.116.132"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-requested-with":["XMLHttpRequest"]}​​​​​}​​​​​}​​​​​,"request":{​​​​​"protocol":"CREST","operation":"READ"}​​​​​,"response":{​​​​​"status":"SUCCESSFUL","statusCode":"","elapsedTime":2,"elapsedTimeUnits":"MILLISECONDS","detail":{​​​​​"objectId":"id=kckadmin,ou=user,dc=openam,dc=forgerock,dc=org","revision":"-1"}​​​​​}​​​​​,"realm":"/","component":"Users"}​​​​​ Thanks Nick ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 16, 2020, 4:38pm UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516/2 "2020-11-16T16:38:37Z")
If your message was valid JSON (which as shown, it is not) then you could do something like ``` json { source => "message" remove_field => ["message"] } ruby { code => ' h = event.get("[http][request][headers]") if h h.each { |k, v| event.set("[headers][#{k}]", v[0]) } end ' } ``` to get ``` "headers" => { "accept-api-version" => "protocol=1.0,resource=2.0", "content-type" => "application/json", "x-forwarded-port" => "443", "x-requested-with" => "XMLHttpRequest", "host" => "openam.dev.pointsection.com", "x-forwarded-for" => "184.143.116.132", "accept" => "application/json, text/javascript, /; q=0.01", "x-forwarded-proto" => "https", "user-agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" } ``` ---
**Author:** ![nikhilesh](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nikhilesh](https://discuss.elastic.co/u/nikhilesh)\ **Post date:** [December 10, 2020, 4:09am UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516/3 "2020-12-10T04:09:50Z")
Thanks Badger, it worked:) Also, can we apply the same json to the below format. I don't think it work for below log. I want to the below log data as separate columns. Please help me out. Thanks in advance. Original log message: [08/Dec/2020:01:36:56 +0000] category=CORE severity=ERROR msgID=200 msg=Entry "uid=test123,ou=people,dc=taphill,dc=com" contains a value "USA" for attribute c that is invalid according to the syntax for that attribute: The provided value "USA" is not a valid country string because the length is not exactly two characters. date: [08/Dec/2020:01:36:56 +0000] or 08/Dec/2020:01:36:56 +0000 category= CORE severity=Error msgID=200 msg=Entry "uid=test123,ou=people,dc=taphill,dc=com" contains a value "USA" for attribute c that is invalid according to the syntax for that attribute: The provided value "USA" is not a valid country string because the length is not exactly two characters Thanks Nick ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 10, 2020, 5:31pm UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516/4 "2020-12-10T17:31:40Z")
It would be specific to that format, but you could do it using ``` kv { include_keys => ["category", "severity", "msgID"] } grok { match => { "message" => "msg=%{GREEDYDATA:msg}" } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 7, 2021, 5:31pm UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516/5 "2021-01-07T17:31:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.