Parsing Json object array

Shawcs · July 17, 2019, 10:15am

Hi,

I have trouble with a dynamic Json file.

My json have an array of "circuitPath". This circuit path is a nested array inside my main json file that can be N element long.

I would like to parse it dynamicaly like so :

this JSON input:

 "circuitPath": [{
		"policy": "policy name 1",
		"execTime": 10,
		"filters": " my first filter"
	}, {
		"policy": "policy name 2",
		"execTime": 20,
		"filters": " my second filter"
	}, {
		"policy": "policy name 3",
		"execTime": 25,
		"filters": " my third filter"
	}
]

would become separeted field in my kibana :

api_policy_name-1=policy name 1
api_policy_Exec_Time-1=10
api_policy_filters-1= my first filter

api_policy_name-2=policy name 2
api_policy_Exec_Time-2=20
api_policy_filters-2=my second filter

api_policy_name-3=policy name 3
api_policy_Exec_Time-3=25
api_policy_filters-3=my third filter

I saw some ruby code to do things looking like that but never found the good one or making it work for my case.
Any information or clues ?

Thanks

Alex

Badger · July 17, 2019, 1:06pm

You could start with

    ruby {
        code => '
            c = event.get("circuitPath")
            c.each_index { |x|
                event.set("api_policy_name-#{x}", c[x]["policy"])
            }
        '
    }

Shawcs · July 17, 2019, 3:42pm

Hi again ,

based on that I got the following error
[ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodeach_index' for
nil:NilClass`

As I understand the parsing fail because my object is empty. So i put it after acontrol like this:

if ([json][circuitPath][0]) {
		ruby {
			code => '
				c = event.get("circuitPath")
				c.each_index { |x|
				event.set("TESTING_api_policy_name-#{x}", c[ x ]["[policy]"])
				}
			'
		}
}

but still I got the error. I'm wondering if this event.get really retrieve my data. Could it be writed like that :
c = event.get("%{[json][circuitPath]}")
or
c = event.get([json][circuitPath]"

this based on how my configuration that work today with this:

mutate {
								add_field => {
									"api_policy_name" => "%{[json][circuitPath][0][policy]}"
									"api_policy_Exec_Time" => "%{[json][circuitPath][0][execTime]}"
									"api_policy_filters" => "%{[json][circuitPath][0][filters]}"
									tags => "circuitPath"
								}
							}

Badger · July 17, 2019, 4:02pm

That is the one I would expect to work. Generally you should add tests in the ruby code to defend against such errors

code => '
    c = event.get("[json][circuitPath]")
    if c
        c.each_index { |x|
            event.set("TESTING_api_policy_name-#{x}", c[ x ]["[policy]"])
        }
    end
'

Shawcs · July 18, 2019, 8:23am

Hi again,

well this is not trowing error anymore but I got no data here is a screen :

For this log the message parsed is the following :

{
	"processInfo": {
		"groupName": "Manager,
		"hostname": "a98sv0",
		"groupId": "group-0",
		"serviceName": "Manager2",
		"serviceId": "instance-2",
		"version": "v7.-Internal",
		"domainId": "3c038a0-a625-a7fa4f04ef9e"
	},
	"circuitPath": [{
			"execTime": 5,
			"policy": "API Broker OAuth 2.0 Security Device",
			"filters": [{
					"name": "<anonymous>",
					"execTime": 5,
					"filterTime": 1563437920625,
					"type": "ValidateOAuthAccessTokenFilter",
					"class": "com.circuit.oauth.provider.ValidateOAuthAccessTokenFilter",
					"status": "Pass"
				}
			]
		}, {
			"execTime": 6701,
			"policy": "API Broker",
			"filters": [{
					"name": "Set service context",
					"execTime": 0,
					"filterTime": 1563437920626,
					"type": "ApiServiceContextFilter",
					"class": "com.coreapireg.runtime.broker.ApiServiceContextFilter",
					"status": "Pass"
				}, {
					"name": "Connect to URL",
					"execTime": 6701,
					"filterTime": 1563437927327,
					"type": "VApiConnectToURLFilter",
					"class": "com.circuit.vapi.VApiConnectToURLFilter",
					"status": "Pass"
				}
			]
		}
	],
	"correlationId": "6020ba34b41e500082",
	"timestamp": 1563437927328
}

and I used this logstash configuration:

							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c
									c.each_index { |x|
										event.set("TESTING_api_policy_name-#{x}", c[x]["[policy]"])
										event.set("TESTING_api_policy_Exec_Time-#{x}", c[x]["[execTime]"])
										event.set("TESTING_api_policy_filters-#{x}", c[x]["[filters]"])
									}
									end
								'
							}

Badger · July 18, 2019, 9:38am

Remove the brackets

event.set("TESTING_api_policy_name-#{x}", c[x]["policy"])
event.set("TESTING_api_policy_Exec_Time-#{x}", c[x]["execTime"])
event.set("TESTING_api_policy_filters-#{x}", c[x]["filters"])

Shawcs · July 18, 2019, 9:57am

Well thanks alot that worked juste fine.

Have a nice day.
See you in my other post ^^

Shawcs · July 18, 2019, 10:51am

well stransgly i still have an error

if I do that

if ([json][circuitPath][0]) {
							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c
									c.each_index { |x|
										event.set("api_policy_name-#{x}", c[x]["policy"])
										event.set("api_policy_Exec_Time-#{x}", c[x]["execTime"])
										event.set("api_policy_filters-#{x}", c[x]["filters"])
									}
									end
								'
							}
							
							mutate {
								add_field => {
									"api_policy_name" => "%{[json][circuitPath][0][policy]}"
									"api_policy_Exec_Time" => "%{[json][circuitPath][0][execTime]}"
									"api_policy_filters" => "%{[json][circuitPath][0][filters]}"
									tags => "circuitPath"
								}
							}
					}

It's work

but if i do that

if ([json][circuitPath][0]) {
							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c
									c.each_index { |x|
										event.set("api_policy_name-#{x}", c[x]["policy"])
										event.set("api_policy_Exec_Time-#{x}", c[x]["execTime"])
										event.set("api_policy_filters-#{x}", c[x]["filters"])
									}
									end
								'
							}
							
							mutate {
								add_field => {
									tags => "circuitPath"
								}
							}
					}

that's not working

Shawcs · August 12, 2019, 9:58am

Hi I still have a question. How can I limit the number of index to 10 for example in this loop?

							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c 
									for c.each_index { |x|
										event.set("api_policy_name-#{x}", c[x]["policy"])
										event.set("api_policy_Exec_Time-#{x}", c[x]["execTime"])
										event.set("api_policy_filters-#{x}", c[x]["filters"])
									}
									end
								'
							}

because I got some errors like that:

[2019-08-12T13:08:41,305][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"api-claranet-dsi-2019.33", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x50a21416>], :response=>{"index"=>{"_index"=>"api-claranet-dsi-2019.33", "_type"=>"doc", "_id"=>"iJKEhWwBD9hy_QDFE2KA", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of mapping depth [20] in index [api-claranet-dsi-2019.33] has been exceeded due to object field [api_policy_filters-0.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths]"}}}}

cheers

Badger · August 12, 2019, 12:31pm

Just test if x is <= 10.

system · September 9, 2019, 12:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Parsing json objects with dynamic key Logstash	3	2206	June 20, 2018
Nested JSON parsing in dynamic field names Logstash	2	2680	July 16, 2018
How to parse dynamic array json through logstash? Logstash	2	386	July 20, 2020
Parsing nested arrays in json objects Logstash	1	358	December 13, 2018
Follow-up from "Unable to avoid JSON parsing errors in logstash log-file" Logstash	1	301	July 20, 2021

Parsing Json object array

Related topics