Logstash Aggregate multi levels

Elastic Stack Logstash
1

Hi all,

So, I am trying do a simple aggregation using plugins-filters-aggregate. I extract data from relation database using plugin-jdbc and the result is:

P_ID	P_DESC 		PROD_I	P_ID	PROD_DESC			PROD_ID SPROD_DESC		COUNTRY
700		ELASTIC		1		700		LOAN DESCRIPTION	1		BLOAN			Mexico
700		ELASTIC		1		700		LOAN DESCRIPTION	2		CLOAN			Chile
700		ELASTIC		2		700		LOAN 2				1		BLOAN			Mexico
700		ELASTIC		2		700		LOAN 2				2		CLOAN			Chile
|--	project --------|----------- product --------------|----sub product -------|-country -|

Where a project has countries and products and products has many sub_products.

I would like this result:

{
	"PROD_I"	: 700,
	"PROD_DESC"	: "ELASTIC",
	"PRODUCTS"	: [
		{
			PROD_ID			: 1,
			PROD_DESC		: "LOAN DESCRIPTION",
			"SUB_PRODUCTS"	: [
				{
					"SPROD_DESC"	: "BLOAN"
				},
				{
					"SPROD_DESC"	: "CLOAN"
				}
			]
		},
		{
			PROD_ID		:	2,
			PROD_DESC	:	"LOAN 2"
		}
	],
	"COUNTRIES"	: [
		{ "NAME" : "Chile"}, 
		{ "NAME":	"Mexico"}
	]
}

I have tried this:

filter {
  aggregate {
    task_id => "%{p_id}"
    code => "
      map['PROD_I'] = event.get('PROD_I')
      map['PROD_DESC'] = event.get('PROD_DESC')

	  map['PRODUCTS'] = []
      map['PRODUCTS'] << {
        'PROD_DESC' =>  event.get('PROD_DESC')
      }
      event.cancel()
    "
    push_previous_map_as_event => true
    timeout => 3
	}
}

How can I improve my aggregation to be the correct project --> products --> sub products?

Best regards,
Ythalo Rossy

5 Likes
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.