# Filtering on nested json fields of varying length

**URL:** <https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373>\
**Category:** Logstash\
**Created:** [March 27, 2020, 9:36am UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373 "2020-03-27T09:36:43Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyoez](https://avatars.discourse-cdn.com/v4/letter/c/c67d28/32.png) [@cyoez](https://discuss.elastic.co/u/cyoez)\
**Post date:** [March 27, 2020, 9:36am UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373/1 "2020-03-27T09:36:44Z")

</div>

Hello everyone,  
First of all, I am new to Logstash, so I would not be surprised if what I am asking is not within the scope of what Logstash can do.  
So, I am working on a project where I need to do multiple GeoIP lookups on a nested json string. Here is what the input string would look like, beautified:

```auto
{
  "testid": "abdc",
  "ipversion": "v4",
  "flow": [
    {
      "destination_port": 33434,
      "dub": [
        {
          "index": 1,
          "ip": "192.168.1.254"
        },
        {
          "index": 2,
          "ip": "82.255.159.254"
        },
        {
          "index": 3,
          "ip": "82.255.210.12"
        }
      ]
    },
    {
      "destination_port": 33438,
      "dub": [
        {
          "index": 1,
          "ip": "172.2.12.12"
        },
        {
          "index": 2,
          "ip": "84.8.15.123"
        }
      ]
    }
  ]
}

```

What I want is to add a GeoIP lookup result field under each "ip" field. I tried to brute-force it by applying filters to every field of the array (even non-existent ones), which worked but was very ugly. I am wondering if there exists a cleaner way to do it by using the Split filter or some other trick I am not yet aware of.

Many thanks in advance,

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 27, 2020, 4:00pm UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373/2 "2020-03-27T16:00:56Z")

</div>

> [@cyoez](#):
>
> I am wondering if there exists a cleaner way to do it by using the Split filter or some other trick I am not yet aware of.

You could add an id field to the event, split each of the arrays, and then aggregate based on the id field, but I am not sure that counts as "cleaner".

---

<div class="post-metadata">

**Author:** ![cyoez](https://avatars.discourse-cdn.com/v4/letter/c/c67d28/32.png) [@cyoez](https://discuss.elastic.co/u/cyoez)\
**Post date:** [March 27, 2020, 4:13pm UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373/3 "2020-03-27T16:13:19Z")

</div>

Thank you for the suggestion, I have been tinkering with a Ruby filter in the meantime, I will go back to your solution if mine is failing. Thank you for your time anyway!

---

<div class="post-metadata">

**Author:** ![cyoez](https://avatars.discourse-cdn.com/v4/letter/c/c67d28/32.png) [@cyoez](https://discuss.elastic.co/u/cyoez)\
**Post date:** [March 27, 2020, 4:44pm UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373/4 "2020-03-27T16:44:05Z")

</div>

Nevermind, I managed to solve my problem with a Ruby filter. I'm leaving my solution here in case it helps someone:

```auto
input {
  file { 
    path => "/path/to/file/file.json"
    codec => "json"
    start_position => "beginning"
  }
}
filter {
   ruby {
      init => "
         require 'logstash/filters/geoip'
      "
      code => "
         nflow = event.get('flow_number') - 1
         for idflow in 0..nflow do
             nhops = event.get(%([flow][#{idflow}])+'[hops_count]') - 1
             for idhop in 0..nhops do
                @geoip = LogStash::Filters::GeoIP.new({'source' => %([flow][#{idflow}][dub][#{idhop}][ip]),
                                       'target' => %([flow][#{idflow}][dub][#{idhop}][geoip])
                                       })

                @geoip.register

                @geoip.filter(event)

             end
         end
      "
   }
}

```

Not the cleanest, but the code is short and completes in a reasonable time.

I'll mark this post as a solution so the topic can be closed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2020, 4:44pm UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373/5 "2020-04-24T16:44:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.