# How to retrieve JSON object from log in logstash confuguration? **URL:** \ **Category:** Logstash\ **Created:** [October 22, 2018, 10:23am UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375 "2018-10-22T10:23:02Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Vyankatesh\_S\_Repal](https://avatars.discourse-cdn.com/v4/letter/v/c77e96/32.png) [@Vyankatesh\_S\_Repal](https://discuss.elastic.co/u/Vyankatesh_S_Repal)\ **Post date:** [October 22, 2018, 10:23am UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375/1 "2018-10-22T10:23:02Z")
I am creating an application where I need to retrieve key value pairs from json object in logs with the help of logstash configuraition. Here is my configuration: ``` input{ file{ path => "D:\ELK_Info\TestLogs_Updated_tablev4.log" start_position => beginning codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } } } filter{ grok{ match => { "message" => "%{IP:client_ip}%{NOTSPACE:space}%{GREEDYDATA:json_data}" } } #mutate { remove_field => ["tags"]} json { source => "json_data" target => "parsedJson" remove_field=>["json_data"]} mutate { add_field => { "AssetManagerId" => "%{[parsedJson][AssetManagerId]}" "Amount" => "%{[parsedJson][Amount]}" "AccountId" => "%{[parsedJson][AccountId]}" "RequestCode" => "%{[parsedJson][RequestCode]}" "TicketNumber" => "%{[parsedJson][TicketNumber]}" "Status" => "%{[parsedJson][Status]}" "message" => ["%{[parsedJson][message]}"] } } } output { file{ path => "D:\ELK_Info\logstashOutput.log" } } ``` Here is the log for example: > Sep 28 15:09:50 52.231.153.246 gateway: [6] INFO AppLog - 180 - XXXGatewayAPI.APIHandlers - UpdateDepositTicket called by xyzadmin from 211.211.211.211: {"AssetManagerId":211,"AccountId":211,"AssetId":211,"AssetName":" ","Amount":"211","RequestCode":"211-211-211-211-211","RequestIP":"211.211.211.211","RequestUser":211,"RequestUserName":"211@211.com","OperatorId":211,"Status":"Accepted","FeeAmt":0,"UpdatedByUser":211,"UpdatedByUserName":"211","TicketNumber":211,"DepositInfo":"{"Full Name":"211","language":"kr","Comments":""}","CreatedTimestamp":"2018-09-27T11:02:22Z","LastUpdateTimeStamp":"211-09-211:09:48.203Z","Comments":,"Attachments":null,"type":"deposit"} With my current configuration, I get key value pair as: > "Status" : "%{[parsedJson][Status]}" whereas, I need exact value for key 'Status' in json in the place of "%{[parsedJson][Status]}". What changes I need to make to have required output? ---
**Author:** ![redX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/redx/32/21864_2.png) [@redX](https://discuss.elastic.co/u/redX)\ **Post date:** [October 22, 2018, 1:58pm UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375/2 "2018-10-22T13:58:53Z")
I do not understand your question. What do you mean with "I need exact value for key 'Status' in json in the place of '%{[parsedJson][Status]}'."? The JSON in the log example seems to be not valid. Formatted it looks like this: ``` { "AssetManagerId": 211, "AccountId": 211, "AssetId": 211, "AssetName": " ", "Amount": "211", "RequestCode": "211-211-211-211-211", "RequestIP": "211.211.211.211", "RequestUser": 211, "RequestUserName": "211@211.com", "OperatorId": 211, "Status": "Accepted", "FeeAmt": 0, "UpdatedByUser": 211, "UpdatedByUserName": "211", "TicketNumber": 211, "DepositInfo": "{" Full Name ":" 211 "," language ":" kr "," Comments ":" "}", "CreatedTimestamp": "2018-09-27T11:02:22Z", "LastUpdateTimeStamp": "211-09-211:09:48.203Z", "Comments": [], "Attachments": null, "type": "deposit" } ``` The stuff in `DepositInfo` is invalid JSON. ---
**Author:** ![Vyankatesh\_S\_Repal](https://avatars.discourse-cdn.com/v4/letter/v/c77e96/32.png) [@Vyankatesh\_S\_Repal](https://discuss.elastic.co/u/Vyankatesh_S_Repal)\ **Post date:** [October 22, 2018, 3:42pm UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375/3 "2018-10-22T15:42:30Z")
Thanks for your correction, redX. What I mean is, currently in output I am getting values as > "AssetManagerId" =\> "%{[parsedJson][AssetManagerId]}" for all fields in json retrieved. But what I need is something like > "AssetManagerId": 211, I need to get value from the input json. Please check json data. Hope you got my point. ---
**Author:** ![redX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/redx/32/21864_2.png) [@redX](https://discuss.elastic.co/u/redX)\ **Post date:** [October 22, 2018, 4:28pm UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375/4 "2018-10-22T16:28:58Z")
Got it. If I remove the `input->file->codec` defintion from your config, it works: config: ``` input { file { path => "C:\ProgramFiles\logstash-6.3.0\test.log" start_position => beginning sincedb_path => "NUL" } } filter { grok { match => { "message" => "%{IP:client_ip}%{NOTSPACE:space}%{GREEDYDATA:json_data}" } } json { source => "json_data" target => "parsedJson" remove_field => ["json_data"] } mutate { add_field => { "AssetManagerId" => "%{[parsedJson][AssetManagerId]}" "Amount" => "%{[parsedJson][Amount]}" "AccountId" => "%{[parsedJson][AccountId]}" "RequestCode" => "%{[parsedJson][RequestCode]}" "TicketNumber" => "%{[parsedJson][TicketNumber]}" "Status" => "%{[parsedJson][Status]}" "message" => ["%{[parsedJson][message]}"] } } } output { stdout { codec => rubydebug } } ``` And I removed the invalid part from your JSON: ``` Sep 28 15:09:50 52.231.153.246 gatewayy: [6] INFO AppLog - 180 - XXXGatewayAPI.APIHandlers - UpdateDepositTicket called by xyzadmin from 211.211.211.211: {"AssetManagerId": 211,"AccountId": 211,"AssetId": 211,"AssetName": " ","Amount": "211","RequestCode": "211-211-211-211-211","RequestIP": "211.211.211.211","RequestUser": 211,"RequestUserName": "211@211.com","OperatorId": 211,"Status": "Accepted","FeeAmt": 0,"UpdatedByUser": 211,"UpdatedByUserName": "211","TicketNumber": 211,"CreatedTimestamp": "2018-09-27T11:02:22Z","LastUpdateTimeStamp": "211-09-211:09:48.203Z","Comments": [],"Attachments": null,"type": "deposit"} ``` Then I get: ``` { "space" => ":", "path" => "C:\\ProgramFiles\\logstash-6.3.0\\test.log", "AccountId" => "211", "host" => "[...]", "Amount" => "211", "@version" => "1", "TicketNumber" => "211", "RequestCode" => "211-211-211-211-211", "client_ip" => "211.211.211.211", "message" => [ [0]"[...]", [1]"%{[parsedJson][message]}" ], "@timestamp" => 2018 - 10 - 22T16: 24: 35.555Z, "Status" => "Accepted", "parsedJson" => { [...] }, "AssetManagerId" => "211" } ``` If your logs are on one line, then you don't need the multiline coded. If you need it, you need to investigate further. But the problem then is with that part only. ---
**Author:** ![Vyankatesh\_S\_Repal](https://avatars.discourse-cdn.com/v4/letter/v/c77e96/32.png) [@Vyankatesh\_S\_Repal](https://discuss.elastic.co/u/Vyankatesh_S_Repal)\ **Post date:** [October 23, 2018, 11:50am UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375/5 "2018-10-23T11:50:59Z")
Thanks redX, it worked. Can you please take a look at this question too? > [@How to retrieve n number of previous lines in log file using logstash configuration?](https://discuss.elastic.co/t/how-to-retrieve-n-number-of-previous-and-after-lines-in-log-file-using-logstash-configuration/153350): > > I am creating an application where I need to put previous and after n number of lines in elasticsearch with current log. This is my current logstash configuraion. What changes I need to make so that I can retrieve last n number of lines(Let's say 5) in my output? input{ file{ path =\> "D:\ELK\_Info\TestLogs\_Updated.log" #start\_position =\> beginning ignore\_older =\> 0 sincedb\_path =\> "NUL" codec =\> multiline { pattern =\> "^%{TIMESTAMP\_ISO8601}" … ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 20, 2018, 11:51am UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375/6 "2018-11-20T11:51:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.