# Parse multi line json **URL:** \ **Category:** Logstash\ **Created:** [July 26, 2020, 11:45pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677 "2020-07-26T23:45:25Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![laxmikanth](https://avatars.discourse-cdn.com/v4/letter/l/db5fbb/32.png) [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Post date:** [July 26, 2020, 11:45pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/1 "2020-07-26T23:45:25Z")
Hi, When I try to parse multi line json as below, I am seeing in ELK as multiline, \_jsonparsefailure, \_grokparsefailure from below json file content, want to remove KEY4 & host and send rest of the fields to elastic search. [ { "KEY1": "ABC", "KEY2": "ABC", "KEY3": "ABC", "KEY4": "{"region":"11","UserSessionId":"222","UserId":"gllexie"}", "host": "ABC", "timestamp": 1595411041516, }, { "KEY1": "ABC2", "KEY2": "ABC2", "KEY3": "ABC2", "KEY4": "{"region":"22","UserSessionId":"No%20CPM%20Profile","UserId":"gllexie"}", "host": "ABC2", "timestamp": 1595411041516, } ] ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [July 27, 2020, 12:54am UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/2 "2020-07-27T00:54:24Z")
Not sure what your current input/filter look like but it would be something like this. ```auto filter { mutate { remove_field => ["host", " KEY4"] } } ``` ---
**Author:** ![laxmikanth](https://avatars.discourse-cdn.com/v4/letter/l/db5fbb/32.png) [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Post date:** [July 27, 2020, 1:42am UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/3 "2020-07-27T01:42:44Z")
filter { mutate { remove\_field =\> ["host", " KEY4"] } } this filter worked, but I see JSON element is in message where as I wanted them to be separate fields in kibana. Can you please help me in resolving this? ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [July 27, 2020, 1:59am UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/4 "2020-07-27T01:59:15Z")
Try this. If it doesn't work can you post your full config? ```auto filter { json { source => "message" remove_field => ["host", " KEY4"] } } ``` ---
**Author:** ![laxmikanth](https://avatars.discourse-cdn.com/v4/letter/l/db5fbb/32.png) [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Post date:** [July 27, 2020, 12:26pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/5 "2020-07-27T12:26:39Z")
input { file { codec =\> multiline { negate =\> true what =\> previous negate =\> true pattern =\> "{\*}" } path =\> ["C:/samplefile.json"] start\_position =\> "beginning" sincedb\_path =\> "C:/logs/dev" } } filter { mutate { remove\_field =\> ["host", " KEY4"] } } output { elasticsearch { hosts =\> "localhost:9200" index =\> "sample-ingest" } stdout { codec =\> rubydebug } } When I use above config, I see all the JSON is sent to kibana as message, where as I want each field in JSON as a separate property in kibana. ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [July 27, 2020, 12:58pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/6 "2020-07-27T12:58:12Z")
If you do the below does it work? Is there a reason you are using multiline? ```auto input { file { path => ["C:/samplefile.json"] start_position => "beginning" sincedb_path => "C:/logs/dev" } } ``` ---
**Author:** ![laxmikanth](https://avatars.discourse-cdn.com/v4/letter/l/db5fbb/32.png) [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Post date:** [July 27, 2020, 4:01pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/7 "2020-07-27T16:01:49Z")
Hi, I am using multiline as my JSON object is spreaded across multiple lines. ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [July 27, 2020, 4:09pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/8 "2020-07-27T16:09:23Z")
Hi there, first of all please make use of the code formatter tool ( ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/6/56aead55ba25111e9209f232f96bf8f16c37a82a.png) ) when pasting non plain text (such as pipeline conf file lines) cause otherwise it'll be more difficult to read and go through. Now, to my understanding you'd like to process that array of json of yours from a file and send each json as a separate document, each one with its fields extracted. Now, first thing I suggest you should do is edit your source file (if possible) to have the whole json on a single line and be careful to leave a empty line at the end of the file. So your file should look something like this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/8/c8f9e27849698a0872c2bce1d64341d71f545f50.png) Also, please note that the one you posted is not a valid json. In fact, pasting it in any json validator, it'll highlight the useless commas after the "timestamp" values and the quotes around the nested KEY4 value. To be a valid json, yours should look something like this: ``` [ { "KEY1": "ABC", "KEY2": "ABC", "KEY3": "ABC", "KEY4": { "region":"11", "UserSessionId":"222", "UserId":"gllexie" }, "host": "ABC", "timestamp": 1595411041516 }, { "KEY1": "ABC2", "KEY2": "ABC2", "KEY3": "ABC2", "KEY4": { "region":"22", "UserSessionId":"No%20CPM%20Profile", "UserId":"gllexie" }, "host": "ABC2", "timestamp": 1595411041516 } ] ``` Now, having said that, what you could do (after you managed to have your json shrinked in one line with a trailing empty line in the file) is a pipeline like the following: ``` input { file { path => "path/to/json/file" start_position => "beginning" sincedb_path => "/dev/null" codec => "json" } } filter { mutate { remove_field => ["host", "KEY4"] } } output { stdout{} } ``` Having your source json file on one line will avoid you that multiline and all the hassles that come with it. Obviously you can replace the standard output with your ES instance. ---
**Author:** ![laxmikanth](https://avatars.discourse-cdn.com/v4/letter/l/db5fbb/32.png) [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Post date:** [July 27, 2020, 5:00pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/9 "2020-07-27T17:00:52Z")
Hi Fabio, Thank you for your response, I need to process KEY4 as string as we have limit on the length of the value of KEY4 so in some objects, KEY4 is not a valid JSON (as it is trimmed once it reaches the limit of length). so, what I am trying to do is remove the KEY4 or convert it as string. ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [July 27, 2020, 5:16pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/10 "2020-07-27T17:16:17Z")
Ok but you need to pass it as a valid json. It means that if you want to treat it as a string, you have to escape the quotes inside the KEY4 value, like so: ``` { "KEY1": "ABC", "KEY2": "ABC", "KEY3": "ABC", "KEY4": "{ \"region\":\"11\", \"UserSessionId\":\"222\", \"UserId\":\"gllexie\" }", "host": "ABC", "timestamp": 1595411041516 } ``` Now this is a valid json and you can use that same approach. Obviously you need to be able to either format the json file properly when writing it (so escaping the quotes in the KEY4 value and put the json as a single line) or to edit it before parsing it with logstash. Can you do that? Otherwise I have to provide you with a slightly less intuitive solution. ---
**Author:** ![laxmikanth](https://avatars.discourse-cdn.com/v4/letter/l/db5fbb/32.png) [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Post date:** [July 27, 2020, 5:54pm UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/11 "2020-07-27T17:54:40Z")
Hi Fabio, Yes, I can format the KEY4 value as `{ \"region\":\"11\", \"UserSessionId\":\"222\", \"UserId\":\"gllexie\" }`, but for some elements it can be `{ \"region\":\"11\", \"UserSessionId\":\"222\", \"UserId\":\"gllexieaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"`, ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [July 28, 2020, 9:57am UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/12 "2020-07-28T09:57:40Z")
Ok so you're telling me some elements might have the value of that field excessively long and the string might not be closed properly with the trailing quotes? If that's the case I have a feeling you need to first grok out the useless KEY4 part and then proceed with the JSON evaluation, otherwise the json won't be a valid one. Also, in case of an excessively long KEY4 value, will you miss the remaining part of the json, too (like host and timestamp)? Can you please post here an example of such a case? Thank you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 25, 2020, 10:00am UTC](https://discuss.elastic.co/t/parse-multi-line-json/242677/13 "2020-08-25T10:00:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.