Aggregate nested arrays

Elastic Stack Logstash
1

Hello everyone
i am new to Elasticsearch, i need help with a use case,
I want to create an array inside an array using the aggregate option

The desired result is to have a list of services, for each service there will be a list of characteristics.
like that :

"services": [
                        {
                            "CreatedAt": "2021-09-28 11:49:51.7283241",
                            "idservice": 1,
                            "CreatedBy": "james.bond",
                            "NomService": "Traiter les cas API de test",
                            "typeServiceApplicatif": "back-end",
                            "Description": "<p>Service pour tester les cas API</p>",
			    **"characteristics " : ["characteristic1","characteristic2","characteristic3"]**
                        },
                        {
                            "CreatedAt": "2021-09-29 09:08:55.2606073",
                            "idservice": 2,
                            "CreatedBy": "bruce.lee",
                            "NomService": "TraiterCasWCF",
                            "typeServiceApplicatif": "back-end",
                            "Description": "<p>un service soap de test</p>",
			 **"characteristics" : ["characteristic1","characteristic2","characteristic3","characteristic4"]**
                        }
                    ],

In the filter I currently have the following code :

        map['services_list']||= []
        map['services']||= []
       
          if (event.get('idservice') != nil)
          if !( map['services_list'].include? event.get('idservice') ) 
           map['services_list'] << event.get('idservice')
            map['services'] << {
                 'idservice' => event.get('idservice'),
                 'NomService' => event.get('nomservice'),
                 'CreatedAt' =>event.get('createdat'),
                 'CreatedBy' => event.get('createdby'),
                 'Description' => event.get('description'),
                 'typeServiceApplicatif' => event.get('typeserviceapplicatif'),
                    }
          end
        end

and I want to add a list of characteristics (characteristics " : ["char1","char2","char3",...] ... etc) for each service.

my SQL query is:

    SELECT 
                  equipe.Id, 
                  equipe.Nom AS NomEquipe, 
                  service_applicatif.NomService,
                  service_applicatif.Id AS IdService,
                  service_applicatif.CreatedAt,
                  service_applicatif.CreatedBy,
                  service_applicatif.Description,
                  type_service_applicatif.Indice As typeServiceApplicatif,
                  balise.Nom AS Characteristic,

Thank's

2

There is an example of building a list inside a list here.

3

Hi Badger,
In my case I am using a filter with the aggregate option like this :

filter {

    aggregate {
      task_id => "%{id}"
      code => "
       
        map['id'] = event.get('id')
        map['NomEquipe'] = event.get('nomequipe')
		
        map['services_list']||= []
        map['services']||= []
       
        if (event.get('idservice') != nil)
          if !( map['services_list'].include? event.get('idservice') ) 
           map['services_list'] << event.get('idservice')
            map['services'] << {
                 'idservice' => event.get('idservice'),
                 'NomService' => event.get('nomservice'),
                 'CreatedAt' =>event.get('createdat'),
                 'CreatedBy' => event.get('createdby'),
                 'Description' => event.get('description'),
                 'typeServiceApplicatif' => event.get('typeserviceapplicatif'),
                    }
          end
        end

        event.cancel()
       "
      push_previous_map_as_event => true
      timeout => 5
    }
	}

in the link you just put the ruby ​​code is used.
is it possible for me to continue with aggregate map ?

4

Yes, the example I linked to does all of the aggregation in ruby. You could do the first level in an aggregate filter, for sure.

5

I am a beginner, could you please help me more with some code !

6

You have not said what your source data looks like, but if you have

{ "createdat": "2021-09-28 11:49:51.7283241", "idservice": 1, "createdby": "james.bond", "nomservice": "Traiter les cas API de test", "typeserviceapplicatif": "back-end", "description": "<p>Service pour tester les cas API</p>", "characteristic": "characteristic1" }
{ "createdat": "2021-09-28 11:49:51.7283241", "idservice": 1, "createdby": "james.bond", "nomservice": "Traiter les cas API de test", "typeserviceapplicatif": "back-end", "description": "<p>Service pour tester les cas API</p>", "characteristic": "characteristic2" }
{ "createdat": "2021-09-28 11:49:52.7283241", "idservice": 2, "createdby": "james.bond", "nomservice": "Foo", "typeserviceapplicatif": "back-end", "description": "Bar", "characteristic": "Baz" }

then if you configure

    mutate { add_field => { "[@metadata][constant]" => 1 } }
    aggregate {
        task_id => "%{[@metadata][constant]}"
        code => '
            map["services"] ||= {}

            idservice = event.get("idservice")
            if idservice
                if ! map["services"][idservice]
                    map["services"][idservice] = {
                        "idservice" => event.get("idservice"),
                        "NomService" => event.get("nomservice"),
                        "CreatedAt" =>event.get("createdat"),
                        "CreatedBy" => event.get("createdby"),
                        "Description" => event.get("description"),
                        "typeServiceApplicatif" => event.get("typeserviceapplicatif"),
                        "characteristics" => []
                    }
                end
                map["services"][idservice]["characteristics"] << event.get("characteristic")
            end
            event.cancel
        '
        push_previous_map_as_event => true
        timeout => 5
    }
    ruby {
        code => '
        # If services is a hash { "key" => "value" } this converts it to [ "key", "value" ]
        services = event.get("services").to_a
        # Next we throw away the keys since those are also inside the value hash
        newServices = []
        services.each { |x|
            newServices << x[1]
        }
        event.set("services", newServices)
        '
    }

you will get

  "services" => [
    [0] {
                    "idservice" => 1,
                    "CreatedAt" => "2021-09-28 11:49:51.7283241",
                  "Description" => "<p>Service pour tester les cas API</p>",
              "characteristics" => [
            [0] "characteristic1",
            [1] "characteristic2"
        ],
                   "NomService" => "Traiter les cas API de test",
                    "CreatedBy" => "james.bond",
        "typeServiceApplicatif" => "back-end"
    },
    [1] {
                    "idservice" => 2,
                    "CreatedAt" => "2021-09-28 11:49:52.7283241",
                  "Description" => "Bar",
              "characteristics" => [
            [0] "Baz"
        ],
                   "NomService" => "Foo",
                    "CreatedBy" => "james.bond",
        "typeServiceApplicatif" => "back-end"
    }
],
7

Thank you Badger

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.