# Logstash filter: aggregate nested arrays
**URL:** https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965
**Category:** Logstash
**Created:** [August 10, 2021, 7:39pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965 "2021-08-10T19:39:27Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)
#### Post date: [August 10, 2021, 7:39pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/1 "2021-08-10T19:39:27Z")
I'm trying to fetch data from MySQL and push it to ElasticSearch using LogStash, although I'm having trouble creating a config file for LogStash that suits my need
I'm `trying to achieve` this result
```auto
{
"products":[
{
"id":1,
"deals":[
{
"id":5,
"options":[
{
"id":3
},
{
"id":8
}
]
}
]
}
]
}
```
In MySQL, each of these has its own table, meaning that
Product -\> Deal `(ONE => MANY)` |
Deal -\> Deal Option`(ONE => MANY)`
To combine them all, I have a MySQL View that would `LEFT JOIN` those tables so I can process everything using LogStash
Here is my current `LogStash Configuration`
```auto
filter {
aggregate {
task_id => "%{id}"
code => "
map['id'] ||= event.get('id')
map['deals'] ||= []
map['deals'] << {'id' => event.get('deal_id')}
event.cancel()
"
push_previous_map_as_event => true
timeout => 15
}
}
```
Although I got stuck at the part where I need to add `Deal Options` to a `Deal`, is my current code correct? If it is, how can I complete it, thanks for your time!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 10, 2021, 7:52pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/2 "2021-08-10T19:52:52Z")
Take a look at [this](https://discuss.elastic.co/t/aggregate-multiple-nested-recursive-logstash/275621/10) thread, which should give you some ideas.
---
### Author:  [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)
#### Post date: [August 10, 2021, 8:19pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/3 "2021-08-10T20:19:48Z")
Sorry, I did already read what's on that thread, I don't quite understand how you exactly did it, can you please elaborate a bit on my use case, it would be really appreciated!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 10, 2021, 8:44pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/4 "2021-08-10T20:44:38Z")
What does your data look like? In JSON notation is it like this?
```auto
{ "product": 1, "deal": 1, "dealOption": 1}
{ "product": 1, "deal": 1, "dealOption": 2}
{ "product": 1, "deal": 2, "dealOption": 75}
{ "product": 14, "deal": 12, "dealOption": 7}
```
---
### Author:  [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)
#### Post date: [August 10, 2021, 9:25pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/5 "2021-08-10T21:25:33Z")
Yes that's correct.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 10, 2021, 9:52pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/6 "2021-08-10T21:52:53Z")
You could try
```
aggregate {
task_id => "%{product}"
code => '
map["id"] ||= event.get("product")
deal = event.get("deal")
map["deals"] ||= []
map["deals"][deal] ||= []
option = event.get("dealOption")
map["deals"][deal][option] ||= []
map["deals"][deal][option] << { "id" => option }
event.cancel()
'
timeout => 5
push_map_as_event_on_timeout => true
timeout_code => '
deals = event.get("deals")
newDeals = []
deals.each_index { |x|
if deals[x]
newOptions = []
deals[x].each_index { |y|
if deals[x][y]
newOptions << { "id" => y }
end
}
newDeals << { "id" => x, "options" => newOptions }
end
}
event.set("deals", newDeals)
'
}
```
That builds map["deals"] as an array of arrays. Most of the entries will be nil. For example, if product 3 has deal 6 and that has options 1 and 2 the map entry will look like
```
{"id"=>3, "deals"=>[nil, nil, nil, nil, nil, nil, [nil, [{"id"=>1}], [{"id"=>2}]]]}
```
The timeout code then iterates through every array and collects all the array indices (ids) that are not nil.
Overall, that will convert
```auto
{ "product": 3, "deal": 6, "dealOption": 1}
{ "product": 3, "deal": 6, "dealOption": 2}
{ "product": 3, "deal": 2, "dealOption": 11}
{ "product": 3, "deal": 2, "dealOption": 18}
{ "product": 14, "deal": 12, "dealOption": 7}
```
into
```auto
{
"deals": [
{
"options": [
{ "id": 11 },
{ "id": 18 }
],
"id": 2
},
{
"options": [
{ "id": 1 },
{ "id": 2 }
],
"id": 6
}
],
"id": 3,
"@version": "1",
"@timestamp": 2021-08-10T21:45:55.935Z
}
```
etc.
---
### Author:  [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)
#### Post date: [August 10, 2021, 10:20pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/7 "2021-08-10T22:20:06Z")
Okay so I started by changing what I had before
```auto
map['deals'] ||= []
map['deals'] << {'id' => event.get('deal_id')}
```
to
```auto
deal = event.get("deal")
map["deals"] ||= []
map["deals"][deal] ||= []
```
and I'm getting an Aggregate exception (no implicit conversion from nil to integer)
---
### Author:  [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)
#### Post date: [August 10, 2021, 10:47pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/8 "2021-08-10T22:47:50Z")
Was able to solve this by adding a few if checks, thank you very much!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 7, 2021, 10:48pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-nested-arrays/280965/9 "2021-09-07T22:48:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.