# KV filter on ugly json log **URL:** \ **Category:** Logstash\ **Created:** [September 16, 2022, 12:46pm UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575 "2022-09-16T12:46:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![stanley783](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@stanley783](https://discuss.elastic.co/u/stanley783)\ **Post date:** [September 16, 2022, 12:46pm UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575/1 "2022-09-16T12:46:27Z")
Hi, received log in json format, json filter parsed correctly but one field, "details", is in a unfinished/incorrect format lets say and json cant handle it, therefore i applied kv but cant get desired outcome - correct parsing. below is mentioned field that is parsed from json as this whole value: `id:"KLJASDJLEIFA" creator:"user" expires:16639 description:"Test4" start_request:{artifacts:"Generic.Info" artifacts:"Generic.DiskSpace" specs:{artifact:"Generic.Info" parameters:{}} specs:{artifact:"Generic.DiskSpace" parameters:{}}} condition:{labels:{label:"test"}}` i tought setting recursive to true would solve my issue but no. Outcome field names are like: ```auto bla.specs.{artifact bla.condition.{labels.{label bla.start_request.{artifacts ``` so parts with brackets are making troubles. in kv filter, among source and target, i have this config: ```auto field_split => " " recursive => true whitespace => "strict" value_split => ":" ``` Also, when i had "include\_brackets =\> true" instead of "recursive" it did not work. Did not try both of them at same time tho. So brackets { } are not considered like they are in json filter. Any ideas how to get desired outcome, if even possible this way? Or would replacing SPACEs with commas, helped and json would be able to do it? My few brain cells are in friday mode already. God bless. ---
**Author:** ![sudhagar\_ramesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhagar_ramesh/32/105673_2.png) [@sudhagar\_ramesh](https://discuss.elastic.co/u/sudhagar_ramesh)\ **Post date:** [September 17, 2022, 7:03am UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575/2 "2022-09-17T07:03:44Z")
Hello @stanley783 you can apply the below for the details field ```auto filter { mutate{ gsub => ["details", "{", ""] gsub => ["details", "}", ""] strip => "details" } kv { source => "details" field_split => " " recursive => true whitespace => "strict" value_split => ":" } ``` Keep Posted on how it goes !!! Thanks !!! ---
**Author:** ![stanley783](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@stanley783](https://discuss.elastic.co/u/stanley783)\ **Post date:** [September 18, 2022, 7:56am UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575/3 "2022-09-18T07:56:27Z")
Hi, thanks for advice but that did not really solved the issue i meant. Removing brackets was not the problem, incorrect parsing logic and field outcome was, when using kv instead of json. To clarify, check the log from part "start\_request" and first two values: `start_request:{artifacts:"Generic.Info" artifacts:"Generic.DiskSpace" specs:{artifact:"Generic.Info" parameters:{}} specs:{artifact:"Generic.DiskSpace" parameters:{}}} condition:{labels:{label:"test"}}` now i get fields without brackets, but other fields that were within/behind the brackets are not considered as part of the previous objects. It parsed as: ```auto details.start_request.artifacts: Generic.Info details.artifacts: Generic.DiskSpace ``` (second one should be also in field/object - details.start\_request.artifacts, same as first one) also later on in the log, it parsed `details.specs.artifact` instead of correct `details.start_request.specs.artifact`. I wanted to try if i can somehow either edit the log to fit json parsing, or use kv filter options to parse it to some correct degree. Thanks. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 18, 2022, 1:25pm UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575/4 "2022-09-18T13:25:21Z")
The issue is that your log is neither a valid JSON nor a valid key-value message, you won't be able to solve this using just those filters. You will probably need to write a custom ruby code and use it to parse it. Does the format of this `details` field changes or it has always the same format, with the keys in the same position? If the keys are always in the same position, then you can solve it using dissect, if the format changes, then I think that the ruby code will be needed. Also, do you have control on the source of this log and can propose changes? ---
**Author:** ![stanley783](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@stanley783](https://discuss.elastic.co/u/stanley783)\ **Post date:** [September 19, 2022, 10:43am UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575/5 "2022-09-19T10:43:07Z")
Correct, ruby was last resort as i am not that skilled, therefore i wanted to verify that it really cant be easily updated other way. Details field change based on action type, and based on how many attributes (artifacts) are included within that action (in this case, 2 - Generic.Info , Generic.DiskSpace). Anyway, i contacted developer/representative of product generating log with this question, and they confirmed it was first intended to be string field, not json (semi-json to help with manual parsing), but they will probably change it to json. In the meantime i probably make some sloppy parsing that will be changed later on. Thanks for advices. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 17, 2022, 10:43am UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575/6 "2022-10-17T10:43:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.