# Aggregate-filter to create nested arrays

**URL:** <https://discuss.elastic.co/t/aggregate-filter-to-create-nested-arrays/115367>\
**Category:** Logstash\
**Created:** [January 12, 2018, 9:20pm UTC](https://discuss.elastic.co/t/aggregate-filter-to-create-nested-arrays/115367 "2018-01-12T21:20:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bosar](https://avatars.discourse-cdn.com/v4/letter/b/a698b9/32.png) [@bosar](https://discuss.elastic.co/u/bosar)\
**Post date:** [January 12, 2018, 9:20pm UTC](https://discuss.elastic.co/t/aggregate-filter-to-create-nested-arrays/115367/1 "2018-01-12T21:20:57Z")

</div>

I'm new to logstash and have been browsing the web for a couple of day for a solution for my problem to create an nested array for a company structure to load into elasticsearch. I want to create an array of companys with a nested array of departments and for each department also an array of groups/sections. I've been copying a lot of code-snippets from the web but I haven't been able to put everything together to get what i want ...

My original source is a MySQL database but I've been captured the JDBC output into a json file. I'll include the wanted output as json files and then the aggregate filter section of my conf-file

It will be much appreciated if someone had the time to help me on my way.

**input.json**  
{  
"company\_id":1,  
"department\_id":1,  
"departments\_section\_id":4,  
"director":"catdoy",  
"department\_name":"Support",  
"name":"Millenium Inc",  
"departments\_section\_manager":"thotom",  
"departments\_section\_name":"Facilities",  
"department\_manager":"jenjon"  
}  
{  
"company\_id":1,  
"department\_id":1,  
"departments\_section\_id":5,  
"director":"catdoy",  
"department\_name":"Support",  
"name":"Millenium Inc",  
"departments\_section\_manager":"bilbre",  
"departments\_section\_name":"IT",  
"department\_manager":"jenjon"  
}  
{  
"company\_id":1,  
"department\_id":2,  
"departments\_section\_id":null,  
"director":"catdoy",  
"department\_name":"Consumer",  
"name":"Millenium Inc",  
"departments\_section\_manager":null,  
"departments\_section\_name":null,  
"department\_manager":"marsmi"  
}  
{  
"company\_id":1,  
"department\_id":3,  
"departments\_section\_id":1,  
"director":"catdoy",  
"department\_name":"Corporate",  
"name":"Millenium Inc",  
"departments\_section\_manager":"petlar",  
"departments\_section\_name":"Finance",  
"department\_manager":"petlar"  
}  
{  
"company\_id":1,  
"department\_id":3,  
"departments\_section\_id":2,  
"director":"catdoy",  
"department\_name":"Corporate",  
"name":"Millenium Inc",  
"departments\_section\_manager":"esmeas",  
"departments\_section\_name":"Legal",  
"department\_manager":"petlar"  
}  
{  
"company\_id":1,  
"department\_id":3,  
"departments\_section\_id":3,  
"director":"catdoy",  
"department\_name":"Corporate",  
"name":"Millenium Inc",  
"departments\_section\_manager":"linpow",  
"departments\_section\_name":"Human Resources",  
"department\_manager":"petlar"  
}  
{  
"company\_id":1,  
"department\_id":4,  
"departments\_section\_id":null,  
"director":"catdoy",  
"department\_name":"Goverment",  
"name":"Millenium Inc",  
"departments\_section\_manager":null,  
"departments\_section\_name":null,  
"department\_manager":"thotom"  
}  
{

```
    "company_id":2,
    "department_id":null,
    "departments_section_id":null,
    "director":"annsma",
    "department_name":null,
    "name":"Nebulose Corp",
    "departments_section_manager":null,
    "departments_section_name":null,
    "department_manager":null
}

```

**wanted-output.json**  
{  
"company\_id":1,  
"director":"catdoy",  
"name":"Millenium Inc",  
"departments": [  
{  
"name":"Support",  
"id":1,  
"manager":"jenjon",  
"sections": [  
{  
"name":"Facilities",  
"id":4,  
"manager":"thotom"  
},  
{  
"name":"IT",  
"id":5,  
"manager":"bilbre"  
}  
]  
},  
{  
"name":"Consumer",  
"id":2,  
"manager":"marsmi",  
"sections": []  
},  
{  
"name":"Corporate",  
"id":3,  
"manager":"petlar",  
"sections":[  
{  
"name":"Finance",  
"id":1,  
"manager":"petlar"  
},  
{  
"name":"Legal",  
"id":2,  
"manager":"esmeas"  
},  
{  
"name":"Human Resources",  
"id":3,  
"manager":"linpow"  
}  
]  
},  
{  
"name":"Goverment",  
"id":4,  
"manager":"thotom",  
"sections":[]  
}  
]  
}  
{  
"company\_id":2,  
"director":"annsma",  
"name":"Nebulose Corp",  
"departments":[]  
}

and the filter part from my **logstash.conf**  
filter {  
aggregate {  
task\_id =\> "%{company\_id}"  
code =\> "  
map['company\_id'] = event.get('company\_id')  
map['name'] = event.get('name')  
map['director'] = event.get('director')  
map['departments\_list'] ||= []  
map['departments'] ||= []  
if (event.get('department\_id') != nil)  
if !( map['departments\_list'].include? event.get('departments\_id') )  
map['departments\_list'] \<\< event.get('department\_id')  
if( event.get('departments\_section\_id') != nil)  
map['departments'] \<\< {  
'id' =\> event.get('department\_id'),  
'name' =\> event.get('department\_name'),  
'manager' =\> event.get('department\_manager'),  
'sections' =\> [] \<\< {  
'id' =\> event.get('departments\_section\_id'),  
'name' =\> event.get('departments\_section\_name'),  
'manager' =\> event.get('departments\_section\_manager')  
}  
}  
else  
map['departments'] \<\< {  
'id' =\> event.get('department\_id'),  
'name' =\> event.get('department\_name'),  
'manager' =\> event.get('department\_manager'),  
'sections' =\> []  
}  
end  
end  
end  
event.cancel()  
"  
push\_previous\_map\_as\_event =\> true  
timeout =\> 5  
}  
mutate {  
remove\_field =\> ["departments\_list"]  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2018, 9:21pm UTC](https://discuss.elastic.co/t/aggregate-filter-to-create-nested-arrays/115367/2 "2018-02-09T21:21:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.