# Logstash grok on dynamic JSON object **URL:** \ **Category:** Logstash\ **Created:** [June 19, 2018, 4:42am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408 "2018-06-19T04:42:56Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![pohsun.teh](https://avatars.discourse-cdn.com/v4/letter/p/f08c70/32.png) [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Post date:** [June 19, 2018, 4:42am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/1 "2018-06-19T04:42:56Z")
Greetings, I have a dynamic JSON object which I would like to extract fields from it This is sample on how the JSON object looks like ``` layers: { tcap: [ { otid: "12:23:34:45" }, { otid: "4h:2f:65:3g", dtid: "h5:f3:v3:h5", }, { otid: "j7:f3:h5:k7" } ] gsm_map: [ { e212_imsi: 45608730 }, { e164_msisdn: 450968345390 }, { } ] } ``` the **TCAP array might be a single object or array of objects** while the content inside might have OTID or DTID. Same goes to the GSM\_MAP following the TCAP **I have few questions to solve my problem:** **1) How do I use grok on dynamic object? It seems right now I have GROKPARSEFAILURE tag if no fields found** **2) From the GSM\_MAP, I might need to traverse the JSON object with regex. Something like "_\_e212\_imsi" or "_\_e164\_msisdn" as long as it ends with the "e212\_imsi" or "e164\_msisdn"** **3) When I traverse for the OTID/DTID, seems I get only part of it. e.g: otid: "12:23:34:45", and my OTID is "12". It doesn't get the full string** **4) If I manage to get my fields, how do I place it on the same hierarchy of JSON object? In this sample, I want to place it on "layers"** Right now, this if what I have which is far from what I wish for ``` grok { match => { "[layers][gsm_map][*_e212_imsi]" => "%{WORD:IMSI}$" "[layers][gsm_map][*_e164_msisdn]" => "%{WORD:MSISDN}$" "[layers][tcap][text_tcap_otid]" => "%{WORD:otid}$" "[layers][tcap][text_tcap_dtid]" => "%{WORD:dtid}$" "[layers][sccp][sccp_called_digits_sccp_digits]" => "%{WORD:sccp_called_digits_sccp_digits}$" "[layers][sccp][sccp_calling_digits_sccp_digits]" => "%{WORD:sccp_calling_digits_sccp_digits}$" } } mutate { add_field => { "layers" => { "gsm_map" => { "imsi" => "%{IMSI}" "msisdn" => "%{MSISDN}" } "tcap" => { "text_tcap_otid" => "%{otid}" "text_tcap_dtid" => "%{dtid}" } "sccp" => { "sccp_called_digits_sccp_digits" => "%{sccp_called_digits_sccp_digits}" "sccp_calling_digits_sccp_digits" => "%{sccp_calling_digits_sccp_digits}" } } } } ``` Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2018, 5:57am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/2 "2018-06-19T05:57:27Z")
Why would you need to use grok in the first place? You already have the values you want in discrete fields. If you don't like the deep hierarchy you can just move the fields around with a mutate filter. ---
**Author:** ![pohsun.teh](https://avatars.discourse-cdn.com/v4/letter/p/f08c70/32.png) [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Post date:** [June 19, 2018, 6:05am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/3 "2018-06-19T06:05:14Z")
You're right somehow, but how do I remove unwanted fields? It is a very big JSON (thousands of fields) which I would like some fields only ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2018, 6:12am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/4 "2018-06-19T06:12:51Z")
If your JSON string is contained in a single field (which it appears to be in your case; `layers`) just copy or move the fields you want and then delete the field containing the rest of the parsed JSON. ---
**Author:** ![pohsun.teh](https://avatars.discourse-cdn.com/v4/letter/p/f08c70/32.png) [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Post date:** [June 19, 2018, 6:24am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/5 "2018-06-19T06:24:39Z")
It works for dynamic objects? Because sometimes the field might be present or not or in array of objects ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2018, 8:37am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/6 "2018-06-19T08:37:40Z")
It depends. You may have to use a ruby filter. ---
**Author:** ![pohsun.teh](https://avatars.discourse-cdn.com/v4/letter/p/f08c70/32.png) [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Post date:** [June 19, 2018, 8:52am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/7 "2018-06-19T08:52:10Z")
That would be last option. Another question; Just found out about prune whitelist/blacklist. Does it able to whitelist/blacklist nested JSON objects? Found out it is unable to filter nested objects but that was old issue (2016). Trying it now and doesn't seem to be able to filter nested objects. Want to get a clear answer thinking might be my wrong configuration. ``` prune { whitelist_names => [ "[^layers$][^frame$][frame_frame_protocols]", "[^layers$][^gsm_map$][e212_imsi]", "[^layers$][^gsm_map$][e164_msisdn]", "[^layers$][^tcap$][text_tcap_otid]", "[^layers$][^tcap$][text_tcap_dtid]", "[^layers$][^sccp$][sccp_called_digits_sccp_digits]", "[^layers$][^sccp$][sccp_calling_digits_sccp_digits]" ] } ``` Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2018, 9:49am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/8 "2018-06-19T09:49:32Z")
> Does it able to whitelist/blacklist nested JSON objects? No. > Found out it is unable to filter nested objects but that was old issue (2016). What does the age of the issue have to do with anything? If the issue is open you can assume that the problem hasn't been fixed. ---
**Author:** ![pohsun.teh](https://avatars.discourse-cdn.com/v4/letter/p/f08c70/32.png) [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Post date:** [June 19, 2018, 9:53am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/9 "2018-06-19T09:53:32Z")
Got it. Guess I got no other option other than the Ruby code. Thanks for your help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2018, 9:53am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408/10 "2018-07-17T09:53:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.