# Parsing Json object array

**URL:** <https://discuss.elastic.co/t/parsing-json-object-array/190954>\
**Category:** Logstash\
**Created:** [July 17, 2019, 10:15am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954 "2019-07-17T10:15:52Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shawcs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shawcs/32/45490_2.png) [@Shawcs](https://discuss.elastic.co/u/Shawcs)\
**Post date:** [July 17, 2019, 10:15am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/1 "2019-07-17T10:15:52Z")

</div>

Hi,

I have trouble with a dynamic Json file.

My json have an array of "circuitPath". This circuit path is a nested array inside my main json file that can be N element long.

I would like to parse it dynamicaly like so :

this JSON input:

```
 "circuitPath": [{
		"policy": "policy name 1",
		"execTime": 10,
		"filters": " my first filter"
	}, {
		"policy": "policy name 2",
		"execTime": 20,
		"filters": " my second filter"
	}, {
		"policy": "policy name 3",
		"execTime": 25,
		"filters": " my third filter"
	}
]

```

would become separeted field in my kibana :

```
api_policy_name-1=policy name 1
api_policy_Exec_Time-1=10
api_policy_filters-1= my first filter

api_policy_name-2=policy name 2
api_policy_Exec_Time-2=20
api_policy_filters-2=my second filter

api_policy_name-3=policy name 3
api_policy_Exec_Time-3=25
api_policy_filters-3=my third filter

```

I saw some ruby code to do things looking like that but never found the good one or making it work for my case.  
Any information or clues ?

Thanks

Alex

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 17, 2019, 1:06pm UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/2 "2019-07-17T13:06:31Z")

</div>

You could start with

```
    ruby {
        code => '
            c = event.get("circuitPath")
            c.each_index { |x|
                event.set("api_policy_name-#{x}", c[x]["policy"])
            }
        '
    }
```

---

<div class="post-metadata">

**Author:** ![Shawcs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shawcs/32/45490_2.png) [@Shawcs](https://discuss.elastic.co/u/Shawcs)\
**Post date:** [July 17, 2019, 3:42pm UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/3 "2019-07-17T15:42:44Z")

</div>

Hi again 😉 ,

based on that I got the following error  
`[ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method`each\_index' for  
nil:NilClass`

As I understand the parsing fail because my object is empty. So i put it after acontrol like this:

```
if ([json][circuitPath][0]) {
		ruby {
			code => '
				c = event.get("circuitPath")
				c.each_index { |x|
				event.set("TESTING_api_policy_name-#{x}", c[x]["[policy]"])
				}
			'
		}
}

```

but still I got the error. I'm wondering if this event.get really retrieve my data. Could it be writed like that :  
`c = event.get("%{[json][circuitPath]}")`  
or  
`c = event.get([json][circuitPath]"`

this based on how my configuration that work today with this:

```
mutate {
								add_field => {
									"api_policy_name" => "%{[json][circuitPath][0][policy]}"
									"api_policy_Exec_Time" => "%{[json][circuitPath][0][execTime]}"
									"api_policy_filters" => "%{[json][circuitPath][0][filters]}"
									tags => "circuitPath"
								}
							}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 17, 2019, 4:02pm UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/4 "2019-07-17T16:02:13Z")

</div>

> [@Shawcs](#):
>
> c = event.get([json][circuitPath]"

That is the one I would expect to work. Generally you should add tests in the ruby code to defend against such errors

```
code => '
    c = event.get("[json][circuitPath]")
    if c
        c.each_index { |x|
            event.set("TESTING_api_policy_name-#{x}", c[x]["[policy]"])
        }
    end
'

```

---

<div class="post-metadata">

**Author:** ![Shawcs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shawcs/32/45490_2.png) [@Shawcs](https://discuss.elastic.co/u/Shawcs)\
**Post date:** [July 18, 2019, 8:23am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/5 "2019-07-18T08:23:54Z")

</div>

Hi again,

well this is not trowing error anymore but I got no data here is a screen :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/1/c12bd8f5069f953ceaa06f94c077d6b1e63ada7b.png)

For this log the message parsed is the following :

```
{
	"processInfo": {
		"groupName": "Manager,
		"hostname": "a98sv0",
		"groupId": "group-0",
		"serviceName": "Manager2",
		"serviceId": "instance-2",
		"version": "v7.-Internal",
		"domainId": "3c038a0-a625-a7fa4f04ef9e"
	},
	"circuitPath": [{
			"execTime": 5,
			"policy": "API Broker OAuth 2.0 Security Device",
			"filters": [{
					"name": "<anonymous>",
					"execTime": 5,
					"filterTime": 1563437920625,
					"type": "ValidateOAuthAccessTokenFilter",
					"class": "com.circuit.oauth.provider.ValidateOAuthAccessTokenFilter",
					"status": "Pass"
				}
			]
		}, {
			"execTime": 6701,
			"policy": "API Broker",
			"filters": [{
					"name": "Set service context",
					"execTime": 0,
					"filterTime": 1563437920626,
					"type": "ApiServiceContextFilter",
					"class": "com.coreapireg.runtime.broker.ApiServiceContextFilter",
					"status": "Pass"
				}, {
					"name": "Connect to URL",
					"execTime": 6701,
					"filterTime": 1563437927327,
					"type": "VApiConnectToURLFilter",
					"class": "com.circuit.vapi.VApiConnectToURLFilter",
					"status": "Pass"
				}
			]
		}
	],
	"correlationId": "6020ba34b41e500082",
	"timestamp": 1563437927328
}

```

and I used this logstash configuration:

```
							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c
									c.each_index { |x|
										event.set("TESTING_api_policy_name-#{x}", c[x]["[policy]"])
										event.set("TESTING_api_policy_Exec_Time-#{x}", c[x]["[execTime]"])
										event.set("TESTING_api_policy_filters-#{x}", c[x]["[filters]"])
									}
									end
								'
							}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 18, 2019, 9:38am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/6 "2019-07-18T09:38:59Z")

</div>

Remove the brackets

```
event.set("TESTING_api_policy_name-#{x}", c[x]["policy"])
event.set("TESTING_api_policy_Exec_Time-#{x}", c[x]["execTime"])
event.set("TESTING_api_policy_filters-#{x}", c[x]["filters"])
```

---

<div class="post-metadata">

**Author:** ![Shawcs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shawcs/32/45490_2.png) [@Shawcs](https://discuss.elastic.co/u/Shawcs)\
**Post date:** [July 18, 2019, 9:57am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/7 "2019-07-18T09:57:36Z")

</div>

Well thanks alot that worked juste fine.

Have a nice day.  
See you in my other post ^^

---

<div class="post-metadata">

**Author:** ![Shawcs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shawcs/32/45490_2.png) [@Shawcs](https://discuss.elastic.co/u/Shawcs)\
**Post date:** [July 18, 2019, 10:51am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/8 "2019-07-18T10:51:04Z")

</div>

well stransgly i still have an error

if I do that

```
if ([json][circuitPath][0]) {
							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c
									c.each_index { |x|
										event.set("api_policy_name-#{x}", c[x]["policy"])
										event.set("api_policy_Exec_Time-#{x}", c[x]["execTime"])
										event.set("api_policy_filters-#{x}", c[x]["filters"])
									}
									end
								'
							}
							
							mutate {
								add_field => {
									"api_policy_name" => "%{[json][circuitPath][0][policy]}"
									"api_policy_Exec_Time" => "%{[json][circuitPath][0][execTime]}"
									"api_policy_filters" => "%{[json][circuitPath][0][filters]}"
									tags => "circuitPath"
								}
							}
					}

```

It's work

but if i do that

```
if ([json][circuitPath][0]) {
							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c
									c.each_index { |x|
										event.set("api_policy_name-#{x}", c[x]["policy"])
										event.set("api_policy_Exec_Time-#{x}", c[x]["execTime"])
										event.set("api_policy_filters-#{x}", c[x]["filters"])
									}
									end
								'
							}
							
							mutate {
								add_field => {
									tags => "circuitPath"
								}
							}
					}

```

that's not working

---

<div class="post-metadata">

**Author:** ![Shawcs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shawcs/32/45490_2.png) [@Shawcs](https://discuss.elastic.co/u/Shawcs)\
**Post date:** [August 12, 2019, 9:58am UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/9 "2019-08-12T09:58:09Z")

</div>

Hi I still have a question. How can I limit the number of index to 10 for example in this loop?

```
							ruby {
								code => '
									c = event.get("[json][circuitPath]")
									if c 
									for c.each_index { |x|
										event.set("api_policy_name-#{x}", c[x]["policy"])
										event.set("api_policy_Exec_Time-#{x}", c[x]["execTime"])
										event.set("api_policy_filters-#{x}", c[x]["filters"])
									}
									end
								'
							}

```

because I got some errors like that:

`[2019-08-12T13:08:41,305][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"api-claranet-dsi-2019.33", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x50a21416>], :response=>{"index"=>{"_index"=>"api-claranet-dsi-2019.33", "_type"=>"doc", "_id"=>"iJKEhWwBD9hy_QDFE2KA", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of mapping depth [20] in index [api-claranet-dsi-2019.33] has been exceeded due to object field [api_policy_filters-0.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths.filters.subPaths]"}}}}`

cheers

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 12, 2019, 12:31pm UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/10 "2019-08-12T12:31:01Z")

</div>

Just test if x is \<= 10.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2019, 12:31pm UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954/11 "2019-09-09T12:31:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.