# Filter Nested Data based on pattern

**URL:** <https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510>\
**Category:** Logstash\
**Created:** [January 15, 2018, 9:35am UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510 "2018-01-15T09:35:00Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [January 15, 2018, 9:35am UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/1 "2018-01-15T09:35:01Z")

</div>

I have incoming JSON data that looks like:  
{ "DataType1": { "system:subsystem" : [{..some nested data fields}]  
"DataType2": { "system:subsystem" : [{..some nested data fields}]  
}

I am trying to separate the events, so they are based on system:subsystem  
I want each of my event to be as of such:  
{system:subsystem  
[{..some nested data fields}]  
}  
system will always be the same, subsystem will differ each time.

whats the best way to extract this from a nested structure?

---

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [January 15, 2018, 1:36pm UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/2 "2018-01-15T13:36:55Z")

</div>

Any ideas? I need to somehow save that "system:subsystem" is a pattern and that based on that,for all of these I will have separate events.  
In my case, subsystem will be always a different value.

Whats the best way to split nested fields into separate events?  
I tried the split filter and no luck. Trying something like:  
split { field =\> "[DataType2][subsystem]" }  
}  
it doesn't work as my subsystem is unique each time. I also tried grok with match but not succeeding with the match.  
Tried something like the below, just to see if I can match the system:subsystem:  
grok {  
match =\> { "message" =\> "%{WORD:DataType}: {%{GREEDYDATA:reeest}"} }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 15, 2018, 10:28pm UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/3 "2018-01-15T22:28:30Z")

</div>

You'll need to use a ruby filter to iterate over the fields and transform them into a single array of objects containing the fields you want to have after the split, followed by a split filter to split the array.

---

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [January 16, 2018, 8:07am UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/4 "2018-01-16T08:07:54Z")

</div>

Can you provide an example of such a configuration for ruby?  
Also, would it be possible to save the "system:subsystem" as key values with kv, instead of using ruby?

---

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [January 16, 2018, 9:36am UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/5 "2018-01-16T09:36:54Z")

</div>

Actually, my data has the following format, bit more complicated. How can I use ruby for this?  
"DataType1" =\> {  
"system:subsystem123" =\> [  
[0] {  
data to be parsed, some in further arrays  
}  
],  
"system:subsystem234" =\> [  
[0] {  
data to be parsed some in further arrays  
}  
],  
"system:subsystem837" =\> [  
[0] {  
data to be parsed some in further arrays  
}  
],  
etc.

Maybe I could use grok/regex to parse field which looks like system:subsystem?

NOTE; I found ruby code that almost does the job, but it doesn't take account the changeable system:subsystem, the field is static.  
filter {  
ruby {  
code =\> "  
event.get('traits').each\_index {|i|  
event.set([traits][i][0], [traits][i][2])  
}  
}  
}  
Any idea how to get the 'traits' in above example to work with a field that is not a static name? For example: "system:subsystem123", "system:subsystem235", "system:subsystem203"?

---

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [January 17, 2018, 1:59pm UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/6 "2018-01-17T13:59:44Z")

</div>

Hi Magnus,

I am trying to attempt with ruby, but its not working- I am getting no output. Can you give advice:  
ruby {  
code =\>  
"event.get[DataType1].each { |e|  
h = {}  
h[SYSTEM:SUBSYSTEM] = e.first  
h.merge! e[1][0]  
event.set[DataType1] \<\< h  
}"  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2018, 2:00pm UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510/7 "2018-02-14T14:00:22Z")

</div>

Closed on February 14, 2018, 2:00pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.