Filter Nested Data based on pattern

[moriol]

I have incoming JSON data that looks like:
{ "DataType1": { "system:subsystem" : [{..some nested data fields}]
"DataType2": { "system:subsystem" : [{..some nested data fields}]
}

I am trying to separate the events, so they are based on system:subsystem
I want each of my event to be as of such:
{system:subsystem
[{..some nested data fields}]
}
system will always be the same, subsystem will differ each time.

whats the best way to extract this from a nested structure?

[moriol]

Any ideas? I need to somehow save that "system:subsystem" is a pattern and that based on that,for all of these I will have separate events.
In my case, subsystem will be always a different value.

Whats the best way to split nested fields into separate events?
I tried the split filter and no luck. Trying something like:
split { field => "[DataType2][subsystem]" }
}
it doesn't work as my subsystem is unique each time. I also tried grok with match but not succeeding with the match.
Tried something like the below, just to see if I can match the system:subsystem:
grok {
match => { "message" => "%{WORD:DataType}: {%{GREEDYDATA:reeest}"} }

[magnusbaeck (Magnus Bäck)]

You'll need to use a ruby filter to iterate over the fields and transform them into a single array of objects containing the fields you want to have after the split, followed by a split filter to split the array.

[moriol]

Can you provide an example of such a configuration for ruby?
Also, would it be possible to save the "system:subsystem" as key values with kv, instead of using ruby?

[moriol]

Actually, my data has the following format, bit more complicated. How can I use ruby for this?
"DataType1" => {
"system:subsystem123" => [
[0] {
data to be parsed, some in further arrays
}
],
"system:subsystem234" => [
[0] {
data to be parsed some in further arrays
}
],
"system:subsystem837" => [
[0] {
data to be parsed some in further arrays
}
],
etc.

Maybe I could use grok/regex to parse field which looks like system:subsystem?

NOTE; I found ruby code that almost does the job, but it doesn't take account the changeable system:subsystem, the field is static.
filter {
ruby {
code => "
event.get('traits').each_index {|i|
event.set([traits][i][0], [traits][i][2])
}
}
}
Any idea how to get the 'traits' in above example to work with a field that is not a static name? For example: "system:subsystem123", "system:subsystem235", "system:subsystem203"?

[moriol]

Hi Magnus,

I am trying to attempt with ruby, but its not working- I am getting no output. Can you give advice:
ruby {
code =>
"event.get[DataType1].each { |e|
h = {}
h[SYSTEM:SUBSYSTEM] = e.first
h.merge! e[1][0]
event.set[DataType1] << h
}"
}