# Aggregate nested arrays

**URL:** <https://discuss.elastic.co/t/aggregate-nested-arrays/285934>\
**Category:** Logstash\
**Created:** [October 5, 2021, 2:17pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934 "2021-10-05T14:17:10Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lyes\_Ouchene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lyes_ouchene/32/77724_2.png) [@Lyes\_Ouchene](https://discuss.elastic.co/u/Lyes_Ouchene)\
**Post date:** [October 5, 2021, 2:17pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/1 "2021-10-05T14:17:10Z")

</div>

Hello everyone  
i am new to Elasticsearch, i need help with a use case,  
I want to create an array inside an array using the aggregate option

The desired result is to have a list of services, for each service there will be a list of characteristics.  
like that :

```auto
"services": [
                        {
                            "CreatedAt": "2021-09-28 11:49:51.7283241",
                            "idservice": 1,
                            "CreatedBy": "james.bond",
                            "NomService": "Traiter les cas API de test",
                            "typeServiceApplicatif": "back-end",
                            "Description": "<p>Service pour tester les cas API</p>",
			    **"characteristics " : ["characteristic1","characteristic2","characteristic3"]**
                        },
                        {
                            "CreatedAt": "2021-09-29 09:08:55.2606073",
                            "idservice": 2,
                            "CreatedBy": "bruce.lee",
                            "NomService": "TraiterCasWCF",
                            "typeServiceApplicatif": "back-end",
                            "Description": "<p>un service soap de test</p>",
			 **"characteristics" : ["characteristic1","characteristic2","characteristic3","characteristic4"]**
                        }
                    ],

```

In the filter I currently have the following code :

```auto
        map['services_list']||= []
        map['services']||= []
       
          if (event.get('idservice') != nil)
          if !( map['services_list'].include? event.get('idservice') ) 
           map['services_list'] << event.get('idservice')
            map['services'] << {
                 'idservice' => event.get('idservice'),
                 'NomService' => event.get('nomservice'),
                 'CreatedAt' =>event.get('createdat'),
                 'CreatedBy' => event.get('createdby'),
                 'Description' => event.get('description'),
                 'typeServiceApplicatif' => event.get('typeserviceapplicatif'),
                    }
          end
        end

```

and I want to add a list of characteristics (characteristics " : ["char1","char2","char3",...] ... etc) for each service.

my SQL query is:

```auto
    SELECT 
                  equipe.Id, 
                  equipe.Nom AS NomEquipe, 
                  service_applicatif.NomService,
                  service_applicatif.Id AS IdService,
                  service_applicatif.CreatedAt,
                  service_applicatif.CreatedBy,
                  service_applicatif.Description,
                  type_service_applicatif.Indice As typeServiceApplicatif,
                  balise.Nom AS Characteristic,

```

Thank's

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 5, 2021, 5:19pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/2 "2021-10-05T17:19:48Z")

</div>

> [@Lyes\_Ouchene](#):
>
> The desired result is to have a list of services, for each service there will be a list of characteristics.

There is an example of building a list inside a list [here](https://discuss.elastic.co/t/split-filebeat-log-with-a-variable-length-and-create-multiple-events-group-by-a-field/269630/2).

---

<div class="post-metadata">

**Author:** ![Lyes\_Ouchene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lyes_ouchene/32/77724_2.png) [@Lyes\_Ouchene](https://discuss.elastic.co/u/Lyes_Ouchene)\
**Post date:** [October 5, 2021, 5:57pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/3 "2021-10-05T17:57:09Z")

</div>

Hi Badger,  
In my case I am using a filter with the aggregate option like this :

```auto
filter {

    aggregate {
      task_id => "%{id}"
      code => "
       
        map['id'] = event.get('id')
        map['NomEquipe'] = event.get('nomequipe')
		
        map['services_list']||= []
        map['services']||= []
       
        if (event.get('idservice') != nil)
          if !( map['services_list'].include? event.get('idservice') ) 
           map['services_list'] << event.get('idservice')
            map['services'] << {
                 'idservice' => event.get('idservice'),
                 'NomService' => event.get('nomservice'),
                 'CreatedAt' =>event.get('createdat'),
                 'CreatedBy' => event.get('createdby'),
                 'Description' => event.get('description'),
                 'typeServiceApplicatif' => event.get('typeserviceapplicatif'),
                    }
          end
        end

        event.cancel()
       "
      push_previous_map_as_event => true
      timeout => 5
    }
	}

```

in the link you just put the ruby ​​code is used.  
is it possible for me to continue with aggregate map ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 5, 2021, 6:55pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/4 "2021-10-05T18:55:05Z")

</div>

> [@Lyes\_Ouchene](#):
>
> in the link you just put the ruby ​​code is used.  
> is it possible for me to continue with aggregate map ?

Yes, the example I linked to does all of the aggregation in ruby. You could do the first level in an aggregate filter, for sure.

---

<div class="post-metadata">

**Author:** ![Lyes\_Ouchene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lyes_ouchene/32/77724_2.png) [@Lyes\_Ouchene](https://discuss.elastic.co/u/Lyes_Ouchene)\
**Post date:** [October 6, 2021, 1:11am UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/5 "2021-10-06T01:11:19Z")

</div>

I am a beginner, could you please help me more with some code !

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 6, 2021, 5:26pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/6 "2021-10-06T17:26:35Z")

</div>

You have not said what your source data looks like, but if you have

```auto
{ "createdat": "2021-09-28 11:49:51.7283241", "idservice": 1, "createdby": "james.bond", "nomservice": "Traiter les cas API de test", "typeserviceapplicatif": "back-end", "description": "<p>Service pour tester les cas API</p>", "characteristic": "characteristic1" }
{ "createdat": "2021-09-28 11:49:51.7283241", "idservice": 1, "createdby": "james.bond", "nomservice": "Traiter les cas API de test", "typeserviceapplicatif": "back-end", "description": "<p>Service pour tester les cas API</p>", "characteristic": "characteristic2" }
{ "createdat": "2021-09-28 11:49:52.7283241", "idservice": 2, "createdby": "james.bond", "nomservice": "Foo", "typeserviceapplicatif": "back-end", "description": "Bar", "characteristic": "Baz" }

```

then if you configure

```
    mutate { add_field => { "[@metadata][constant]" => 1 } }
    aggregate {
        task_id => "%{[@metadata][constant]}"
        code => '
            map["services"] ||= {}

            idservice = event.get("idservice")
            if idservice
                if ! map["services"][idservice]
                    map["services"][idservice] = {
                        "idservice" => event.get("idservice"),
                        "NomService" => event.get("nomservice"),
                        "CreatedAt" =>event.get("createdat"),
                        "CreatedBy" => event.get("createdby"),
                        "Description" => event.get("description"),
                        "typeServiceApplicatif" => event.get("typeserviceapplicatif"),
                        "characteristics" => []
                    }
                end
                map["services"][idservice]["characteristics"] << event.get("characteristic")
            end
            event.cancel
        '
        push_previous_map_as_event => true
        timeout => 5
    }
    ruby {
        code => '
        # If services is a hash { "key" => "value" } this converts it to ["key", "value"]
        services = event.get("services").to_a
        # Next we throw away the keys since those are also inside the value hash
        newServices = []
        services.each { |x|
            newServices << x[1]
        }
        event.set("services", newServices)
        '
    }

```

you will get

```
  "services" => [
    [0] {
                    "idservice" => 1,
                    "CreatedAt" => "2021-09-28 11:49:51.7283241",
                  "Description" => "<p>Service pour tester les cas API</p>",
              "characteristics" => [
            [0] "characteristic1",
            [1] "characteristic2"
        ],
                   "NomService" => "Traiter les cas API de test",
                    "CreatedBy" => "james.bond",
        "typeServiceApplicatif" => "back-end"
    },
    [1] {
                    "idservice" => 2,
                    "CreatedAt" => "2021-09-28 11:49:52.7283241",
                  "Description" => "Bar",
              "characteristics" => [
            [0] "Baz"
        ],
                   "NomService" => "Foo",
                    "CreatedBy" => "james.bond",
        "typeServiceApplicatif" => "back-end"
    }
],
```

---

<div class="post-metadata">

**Author:** ![Lyes\_Ouchene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lyes_ouchene/32/77724_2.png) [@Lyes\_Ouchene](https://discuss.elastic.co/u/Lyes_Ouchene)\
**Post date:** [October 19, 2021, 9:17pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/7 "2021-10-19T21:17:33Z")

</div>

Thank you Badger

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 16, 2021, 9:17pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934/8 "2021-11-16T21:17:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.