# Nested Json Array parse failure **URL:** \ **Category:** Logstash\ **Created:** [September 30, 2020, 7:57pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564 "2020-09-30T19:57:21Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Feedy](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Post date:** [September 30, 2020, 7:57pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/1 "2020-09-30T19:57:21Z")
I am having trouble parsing json log file that contains nested array. I am trying to add the values in the array after "docs" as their own searchable field linked to the target docs. Here is sample log below: ```auto {"dba_server":"dbaserver","dba_version":"123.4.0.1123610.1729","docs":[{"childproc_count":0,"cmdline":"psexec.exe -i -s C:\\WINDOWS\\system32\\mmc.exe /s taskschd.msc","crossproc_count":1,"filemod_count":4,"host_type":"workstation","last_update":"2020-09-30T11:43:58.808Z","modload_count":52,"netconn_count":0,"os_type":"windows","parent_guid":"00013235-0000-2c90-01d6-971ee9af8eba","parent_name":"cmd.exe","parent_pid":11408,"parent_segment_id":"1","parent_unique_id":"00013235-0000-2c90-01d6-971ee9af8eba-000000000001","path":"c:\\users\\xxxx\\documents\\2-work\\c-programs\\pstools\\psexec.exe","process_guid":"00013235-0000-15f4-01d6-971efba829ba","process_md5":"27304B246C7D5B4E149124D5F93C5B01","process_name":"psexec.exe","process_pid":5620,"process_sha256":"3337e3875b05e0bfba69ab926532e3f179e8cfbf162ebb60ce58a0281437a7ef","regmod_count":1,"segment_id":"1601466945474","start":"2020-09-30T11:43:58.092Z","unique_id":"1234asdf35-0000-15f4-01d6-97asdfba829ba-0174dasdf4fc2","username":"asdf\\xxxx","watchlist_1797":"2020-09-30T11:50:03.222527Z","watchlist_tag":"1797|asdf.suspicious.asdf"}],"process_guid":"00013235-0000-15f4-01d116-9712341efbasdf9asdf","process_id":"00013235-0000-15f4-01d6-971efba829ba","segment_id":"1","server_name":"xx-xxx.xxx.edu","timestamp":1601466945.474,"type":"watchlist.storage.hit.process","watchlist_id":1711234,"watchlist_name":"xxx.asdf.asdf"} ``` Here is my config file(I've tried several different changes but this is the latest): ```auto filter { if [event_type] == "asdf" { grok { match => { "message" => '{"dba_server":"%{WORD:dba_server}","dba_version":"%{DATA:dba_version}","docs":\[{"%{GREEDYDATA:msg}"}\],"%{GREEDYDATA:msg1}"}'} } json { source => "msg" target => "docs" remove_field => "msg" } json { source => "msg1" remove_field => "msg1" } } } ``` This config results in jsonparsefailure. Any ideas on what I am doing wrong here? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 30, 2020, 8:22pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/2 "2020-09-30T20:22:41Z")
Neither [msg] nor [msg1] are valid JSON, since they are not surrounded by {} Why not just parse the whole message? ``` filter { json { source => "message" remove_field => ["message"] } } ``` ---
**Author:** ![Feedy](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Post date:** [September 30, 2020, 8:28pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/3 "2020-09-30T20:28:44Z")
@Badger What you suggested was my first attempt and even though I don't get a jsonparsefailure, the parsed key:values doesn't show in Kibana as an available field but what's confusing is that I can see the key:values as filter options in the search field in Kibana. Is there a particular reason why that is happening? Also thanks for the reply ---
**Author:** ![Feedy](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Post date:** [September 30, 2020, 8:53pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/4 "2020-09-30T20:53:04Z")
Here is a few screenshots of what I am referring to. First Below is the updated conf file.... ```auto filter { if [event_type] == "abc" { json { source => "message" remove_field => ["timestamp", "message"] } } } ``` Here is the output in Kibana of the available fields but not the fields in the array: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/5/455c4ba9aa12c40c9c4b6fa6dff7c7a777c0227e.png) Here is a screenshot of the expanded document that shows the array under the docs field: ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/7/8785104d0e8dd2fe33041ae1af95bd0930b96148.png) And here is a screenshot of the search field showing the filters: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/e/4e97cb3d03b84a9e067901385f7b04e9303e07aa.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 30, 2020, 9:14pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/5 "2020-09-30T21:14:36Z")
Neither "key" nor "value" appears in your sample JSON, so I do not understand why you would expect them to appear in the event. ---
**Author:** ![Feedy](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Post date:** [September 30, 2020, 9:36pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/6 "2020-09-30T21:36:31Z")
Maybe I am using the wrong words and my apologies if that's the case. I am not saying the actual word "key" or "value" is missing, what I am referring to, for example, from this sample log file: ```auto {"dba_server":"dbaserver","dba_version":"6.4.0.1asdf0.17234","docs":[{"childproc_count":0,"cmdline":"fltmc unload paritydriver","comms_ip":"xx.xxx.xx.xx","computer_name":"xxxx-xxxx-xxx","crossproc_count":1,"emet_count":0,"filemod_count":0,"filtering_known_dlls":false,"group":"default group","host_type":"asdf","hostname":"xxxx-xxxx-xxx","id":"00xxxxb-00asdf01210-0e1asdfc-01d6-91f655def5e8","interface_ip":"xx.xxx.xx.xx","last_server_update":-1,"last_update":"2020-09-23T22:10:24.355Z","modload_count":28,"netconn_count":0,"os_type":"xxx","parent_guid":"0002479b-0000-06f4-01d6-91db3a5ec28d","parent_id":"0002479b-0000-06f4-01d6-91db3a5ec28d","parent_md5":"00000000000000000000000000000000","parent_name":"cmd.exe","parent_pid":1780,"parent_segment_id":"1","parent_unique_id":"0002479b-0000-06f4-01d6-91db3a5ec28d-000000000001","path":"c:\\windows\\system32\\xxx.exe","process_guid":"000asdf2asdfaa4asfd79b-0000-0e1c-01d6-91f655def5e8","process_md5":"BCACasdfasdfASDFASFREWTYFGEED1DCE5619C63D21F7112","process_name":"xxxmc.exe","process_pid":3612,"process_sha256":"@!#$HNJKZX94530ASDF8c1220319cbb69e2a2370334d7849304eb3bb7b1d2fd811c4324e","processblock_count":0,"regmod_count":0,"segment_id":"1601497621219","sensor_id":149403,"start":"2020-09-23T22:10:24.266Z","unique_id":"0002479b-0000-0e1c-01d6--0174e0b262e3","username":"xxxx\\xxxx"}],"highlights_by_doc":{"0002479b-0000-0e1c-01d6-91f655def5e8-0174e0b262e3":["xxxx PREPREPREunloadPOSTPOSTPOST paritydriver","c:\\windows\\system32\\PREPREPRExxxx.exePOSTPOSTPOST"]},"server_name":"xxx-xxxx","timestamp":1601498166.457215,"type":"watchlist.xxx.xxx","watchlist_id":1111,"watchlist_name":"xxxx"} ``` I am not seeing the values from the array in the available fields area in Kibana but I can search for those fields via KQL. Take the "comms\_ip" from the sample log...I can search against this field and get the correct documents... ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/6/1637509182046aff8b6508c42c75d4a8366b731d.png) But I don't see the "docs.comm\_ip" in the available fields area ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/9/e94f1ee7bb9fc8b8e7594fb8b6cadd2d131e2061.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 30, 2020, 10:20pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/7 "2020-09-30T22:20:50Z")
Did you refresh the index pattern in Kibana? ---
**Author:** ![Feedy](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Post date:** [September 30, 2020, 11:07pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/8 "2020-09-30T23:07:14Z")
Yes I actually deleted the index before update filter conf file and I also refreshed pattern but still not seeing them in available fields. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 30, 2020, 11:16pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/9 "2020-09-30T23:16:04Z")
I wonder if that is related to the fact that docs is an array. You could try either ``` mutate { replace => { "docs" => "%{[docs][0]}" } } ``` or ``` split { field => "docs" } ``` to see if it then works the way you want. If it does then you would be in a position to ask a question around arrays in the kibana forum. ---
**Author:** ![Feedy](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Post date:** [October 2, 2020, 9:17pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/10 "2020-10-02T21:17:57Z")
Finally got a chance to revisit this and your 2nd suggestion(split filter) is what solved my problem. Thanks again for your help as always. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 30, 2020, 9:18pm UTC](https://discuss.elastic.co/t/nested-json-array-parse-failure/250564/11 "2020-10-30T21:18:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.