Logstash grok on dynamic JSON object

Elastic Stack Logstash
1

Greetings,

I have a dynamic JSON object which I would like to extract fields from it

This is sample on how the JSON object looks like

layers: {
   tcap: [
   {
       otid: "12:23:34:45"
   },
   {
       otid: "4h:2f:65:3g",
       dtid: "h5:f3:v3:h5",
   },
   {
       otid: "j7:f3:h5:k7"
   }
   ]
   gsm_map: [
   {
       e212_imsi: 45608730
   },
   {
       e164_msisdn: 450968345390
   },
   {

   }
   ]
}

the TCAP array might be a single object or array of objects while the content inside might have OTID or DTID.
Same goes to the GSM_MAP following the TCAP

I have few questions to solve my problem:
1) How do I use grok on dynamic object? It seems right now I have GROKPARSEFAILURE tag if no fields found
2) From the GSM_MAP, I might need to traverse the JSON object with regex. Something like "_e212_imsi" or "_e164_msisdn" as long as it ends with the "e212_imsi" or "e164_msisdn"
3) When I traverse for the OTID/DTID, seems I get only part of it. e.g: otid: "12:23:34:45", and my OTID is "12". It doesn't get the full string
4) If I manage to get my fields, how do I place it on the same hierarchy of JSON object? In this sample, I want to place it on "layers"

Right now, this if what I have which is far from what I wish for

grok {
		match => {
			"[layers][gsm_map][*_e212_imsi]" => "%{WORD:IMSI}$"
			"[layers][gsm_map][*_e164_msisdn]" => "%{WORD:MSISDN}$"
			"[layers][tcap][text_tcap_otid]" => "%{WORD:otid}$"
			"[layers][tcap][text_tcap_dtid]" => "%{WORD:dtid}$"
			"[layers][sccp][sccp_called_digits_sccp_digits]" => "%{WORD:sccp_called_digits_sccp_digits}$"
			"[layers][sccp][sccp_calling_digits_sccp_digits]" => "%{WORD:sccp_calling_digits_sccp_digits}$"
		}
	}

mutate {
	add_field => {
		"layers" => {
			"gsm_map" => {
				"imsi" => "%{IMSI}"
				"msisdn" => "%{MSISDN}"
			}
			"tcap" => {
				"text_tcap_otid" => "%{otid}"
				"text_tcap_dtid" => "%{dtid}"
			}
			"sccp" => {
				"sccp_called_digits_sccp_digits" => "%{sccp_called_digits_sccp_digits}"
				"sccp_calling_digits_sccp_digits" => "%{sccp_calling_digits_sccp_digits}"
			}
		}
	}
}

Thanks in advance.

2

Why would you need to use grok in the first place? You already have the values you want in discrete fields. If you don't like the deep hierarchy you can just move the fields around with a mutate filter.

3

You're right somehow, but how do I remove unwanted fields? It is a very big JSON (thousands of fields) which I would like some fields only

4

If your JSON string is contained in a single field (which it appears to be in your case; layers) just copy or move the fields you want and then delete the field containing the rest of the parsed JSON.

5

It works for dynamic objects? Because sometimes the field might be present or not or in array of objects

6

It depends. You may have to use a ruby filter.

7

That would be last option.

Another question; Just found out about prune whitelist/blacklist.
Does it able to whitelist/blacklist nested JSON objects?
Found out it is unable to filter nested objects but that was old issue (2016).

Trying it now and doesn't seem to be able to filter nested objects. Want to get a clear answer thinking might be my wrong configuration.

prune {
 		whitelist_names => [ 
 			"[^layers$][^frame$][frame_frame_protocols]",
			"[^layers$][^gsm_map$][e212_imsi]",
 			"[^layers$][^gsm_map$][e164_msisdn]",
 			"[^layers$][^tcap$][text_tcap_otid]",
 			"[^layers$][^tcap$][text_tcap_dtid]", 
 			"[^layers$][^sccp$][sccp_called_digits_sccp_digits]",
 			"[^layers$][^sccp$][sccp_calling_digits_sccp_digits]"
 		]
 	}

Thanks

8

Does it able to whitelist/blacklist nested JSON objects?

No.

Found out it is unable to filter nested objects but that was old issue (2016).

What does the age of the issue have to do with anything? If the issue is open you can assume that the problem hasn't been fixed.

9

Got it. Guess I got no other option other than the Ruby code.

Thanks for your help.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.