# I would like to access nested data from an array

**URL:** <https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432>\
**Category:** Logstash\
**Created:** [January 12, 2017, 8:45pm UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432 "2017-01-12T20:45:50Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mathias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mathias/32/23176_2.png) [@mathias](https://discuss.elastic.co/u/mathias)\
**Post date:** [January 12, 2017, 8:45pm UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/1 "2017-01-12T20:45:50Z")

</div>

I have a data structure looking like this:

printed as codec =\> rubydebug  
{  
"sequence\_number" =\> 557,  
"event\_tags" =\> [  
[0] {  
"value" =\> "nacl-st-qkrrfq-ppu-server-6y35vxntjont",  
"key" =\> "originhost"  
}  
]  
.  
.  
.  
.  
}

or printed as codec =\> line  
"event\_tags":[{"value":"nacl-st-alcags-ppu-server-zwul4nphc2lc","key":"originhost"}]

or from kibana:  
event\_tags {  
"value": "nacl-st-ibgki7-ppu-server-bl46vcsakx5f",  
"key": "originhost"  
}

I would like ta add a field calld "originhost" which has the value "nacl-st-alcags-ppu-server-zwul4nphc2lc" in this example.

I have tried the following in a mutate filter:  
add\_field =\> {"originhost" =\> "%{[0][event\_tags][key]}"}  
and  
add\_field =\> {"originhost" =\> "%{[event\_tags][key]}"}

but it does not work.

The second one causes java errors in logstash log and the first one "originhost = %{[0][event\_tags][key]}"

How should I write the filter?

Br Mathias

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [January 13, 2017, 10:10am UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/2 "2017-01-13T10:10:00Z")

</div>

if your event structure is:

```auto
{
  "sequence_number" => 557,
  "event_tags" => [
    [0] {
      "value" => "nacl-st-qkrrfq-ppu-server-6y35vxntjont",
      "key" => "originhost"
    }
  ]
}

```

And you want to extract the array element where the key is "originhost" then you need the ruby filter.  
This is because there's no way of knowing if there is more than on element with originhost (or none)  
With ruby filter, you can do:

```auto
filter { ruby { code => "event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']" } }

```

---

<div class="post-metadata">

**Author:** ![mathias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mathias/32/23176_2.png) [@mathias](https://discuss.elastic.co/u/mathias)\
**Post date:** [January 17, 2017, 10:48am UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/3 "2017-01-17T10:48:57Z")

</div>

Hi Juao,

Thanks for your reply.  
I still have problems to make the ruby filter have any affect.  
I do not see any new string field called originhost.

My filter config looks like this:

if "logstash-eventbus-tcp" in [tags] {  
mutate {  
remove\_tag =\> ["logstash-eventbus-tcp"]  
rename =\> { "type" =\> "eventtype" }  
add\_field =\> { "type" =\> "event/%{[eventtype]}" }  
}

mutate {  
rename =\> { "tags" =\> "event\_tags" }  
add\_field =\> { "originhostX" =\> "%{[event\_tags]}" }  
}

ruby { code =\> "event.get('event\_tags').find {|h| h['key'] == 'originhost'}['value']" }  
}

The kibana json format of an event looks like this:  
{  
"\_index": "events-2017.01.17",  
"\_type": "event/system/counter/pbuf",  
"\_id": "AVmsB530vNaxVEhbQd0d",  
"\_score": null,  
"\_source": {  
"sequence\_number": 881,  
"event\_tags": [  
{  
"value": "nacl-st-qm8jef-ppu-server-sy4b5tjkl3iz",  
"key": "originhost"  
}  
],  
"@timestamp": "2017-01-17T10:44:57.666Z",  
"originhostX": "{value=nacl-st-qm8jef-ppu-server-sy4b5tjkl3iz, key=originhost}",  
"data": {  
"resources": [  
{  
"identity": "1",  
"type": "ue\_bearer\_id"  
},  
{  
"identity": "bearer\_deleted",  
"type": "name"  
},  
{  
"identity": "2017-01-17 11:44:57.617778",  
"type": "time"  
}  
]  
},  
"port": 59873,  
"@version": "1",  
"host": "10.68.32.209",  
"producer": "6df18e784e60412830f676ed811bc6d5c8721e82",  
"eventtype": "system/counter/pbuf",  
"type": "event/system/counter/pbuf",  
"timestamp": "2017-01-17T10:44:57.937752962Z"  
},  
"fields": {  
"timestamp": [  
1484649897937  
],  
"@timestamp": [  
1484649897666  
]  
},  
"sort": [  
1484649897666  
]  
}

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [January 17, 2017, 11:05am UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/4 "2017-01-17T11:05:52Z")

</div>

there is one thing missing from that code, which is setting the retrieved value back into the event:

so while this code has no side effect because it only retrieves the data:

```auto
ruby { code => "event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']" }

```

to the value back into the event you need

```auto
ruby { code => "value = event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']; event.set('originhost', value)" }

```

---

<div class="post-metadata">

**Author:** ![mathias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mathias/32/23176_2.png) [@mathias](https://discuss.elastic.co/u/mathias)\
**Post date:** [January 17, 2017, 1:05pm UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/5 "2017-01-17T13:05:21Z")

</div>

> [@jsvd](#):
>
> ruby { code =\> "value = event.get('event\_tags').find {|h| h['key'] == 'originhost'}['value']; event.set('originhost', value)" }

Many thanks to you Joao.

It works perfectly!

Must ask you right away. Is it possible to make a generic filter which creates fields of each key with it's corresponding value without having to specify each key?  
For example from the following data structure:  
{  
"\_index": "events-2017.01.17",  
"\_type": "event/system/counter/pbuf",  
"\_id": "AVmsadkeP1\_tDGOJZ1RF",  
"\_score": null,  
"\_source": {  
"data": {  
"counters": [  
{  
"name": "n\_lost\_packets",  
"value": 0  
},  
{  
"name": "n\_sent\_bytes",  
"value": 130852808  
},  
{  
"name": "n\_received\_packets",  
"value": 51705  
},  
{  
"name": "n\_sent\_packets",  
"value": 94488  
},  
{  
"name": "n\_ce\_bytes",  
"value": 0  
},  
{  
"name": "n\_received\_bytes",  
"value": 2883215  
}  
],

I would like to see 6 fields with the names n\_lost\_packets, n\_sent\_bytes, n\_received\_packets, n\_sent\_packets, n\_ce\_bytes and n\_received\_bytes.

What would the find and set command look like for this task?

Thanks  
Mathias

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [January 17, 2017, 1:29pm UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/6 "2017-01-17T13:29:06Z")

</div>

I think the major issue here is that the source application is sending essentially a hash in array form.  
If the counters element was:

```auto
"counters": {
  "n_lost_packets": 0,
  "n_sent_bytes": 130852808
}

```

instead of:

```auto
"counters": [
  {
    "name": "n_lost_packets",
    "value": 0
  },
  {
    "name": "n_sent_bytes",
    "value": 130852808
  }
]

```

it would be simpler. If you want to programatically you need ruby filter since, in your case each hash contains a key under "key" and a value under "value", but it could be something else, also there can be duplicates, some elements may only contain key or value (errors happen), etc

in your case, to iterate over counters and set each property on event, you can do:

```auto
filter { ruby { code => "event.get('counters').each {|hash| event.set(hash['key'] = hash['value']) }" } }

```

(this code wasn't tested)

---

<div class="post-metadata">

**Author:** ![mathias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mathias/32/23176_2.png) [@mathias](https://discuss.elastic.co/u/mathias)\
**Post date:** [January 18, 2017, 7:21am UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/7 "2017-01-18T07:21:57Z")

</div>

Thanks Joao,

From the latest filter I get an ruby exception:  
[2017-01-18T06:50:23,571][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClas

In Kibana I can see that there is an exception tag called "\_rubyexception" added.

```
{
  "_index": "events-2017.01.18",
  "_type": "event/system/counter/pbuf",
  "_id": "AVmwVgFvpR5aTmxaCXPP",
  "_score": null,
  "_source": {
    "sequence_number": 568,
    "event_tags": [],
    "@timestamp": "2017-01-18T06:49:01.895Z",
    "data": {
      "counters": [
        {
          "name": "n_lost_packets",
          "value": 0
        },
        {
          "name": "n_sent_bytes",
          "value": 474496416
        },
        {
          "name": "n_received_packets",
          "value": 176983
        },
        {
          "name": "n_sent_packets",
          "value": 339238
        },
        {
          "name": "n_ce_bytes",
          "value": 0
        },
        {
          "name": "n_received_bytes",
          "value": 9745601
        }
      ],
      "resources": [
        {
          "identity": "bearer_path_pm",
          "type": "name"
        },
        {
          "identity": "1",
          "type": "ue_bearer_id"
        },
        {
          "identity": "",
          "type": "pm_id"
        },
        {
          "identity": "3",
          "type": "path_identifier"
        },
        {
          "identity": "2017-01-18 07:49:02.267653",
          "type": "time"
        }
      ]
    },
    "port": 37438,
    "@version": "1",
    "host": "10.68.32.232",
    "producer": "ddc3eb8dfc39e8b785983a0cc57848c5621d1e08",
    "eventtype": "system/counter/pbuf",
    "type": "event/system/counter/pbuf",
    "timestamp": "2017-01-18T06:49:03.016452074Z",
    "tags": [
      "_rubyexception"
    ]
  },
  "fields": {
    "timestamp": [
      1484722143016
    ],
    "@timestamp": [
      1484722141895
    ]
  },
  "sort": [
    1484722141895
  ]
}

```

I have tried to check if the event is nil or not and that prevents the exception.  
ruby { code =\> "myArr = event.get('data.counters')  
if myArr  
event.get('data.counters').each {|hash| event.set(hash['key'] = hash['value']) }  
end" }

I have tried both myArr = event.get('counters') and myArr = event.get('data.counters'). It gives the same result. No action.

Any ideas what is wrong?

Kind regards  
Mathias

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 15, 2017, 7:22am UTC](https://discuss.elastic.co/t/i-would-like-to-access-nested-data-from-an-array/71432/8 "2017-02-15T07:22:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.