Logstash aggregation filter

Elastic Stack Logstash
1

Hi all,
I am trying to aggregate information of multi events to one event using aggregation filter of logstash. for example events are as following:

name1=c1,name2=s1
name1=c1,name2=s1
name1=c1,name2=s1
name1=c1,name2=s2
name1=c1,name2=s2
name1=c2,name2=s1
name1=c2,name2=s1

the expected output should be as following so that shows count of fields:
c1,s1,3
c1,s2,2
c2,s1,2

for this purpose i used following aggregation filter:

aggregate {
    task_id => "%{%{name1}_%{name2}}"
    code => "
		          map['count'] ||= 0


		 		 map['count'] +=1

		 "

		 }

whereas the output is as following:

c1,s1,1
c1,s1,2
c1,s1,3
c1,s2,1
c1,s2,2
c2,s1,1
c2,s1,2

How can i solve this issue?
any advise will be so appreciated.

2

The aggregate filter is good as far as it goes, but nothing will ever trigger it to push the data from the map as an event. If the data is sorted you can do something like example 4. If it is not sorted you can use a timeout, like example 3, but make sure you disable java_execution!

3

I set "pipeline.java_execution: false" in the logstash.yml and change aggregate filter as following:

aggregate {
    task_id => "%{aggregate_id}"
    code => "
		          map['company_count'] ||= 0


		 		 map['company_count'] +=1



		 event.cancel()
		 "
	     push_map_as_event_on_timeout => true

         timeout => 180
		 }

but still output is same as last comment. it is noted that output is not sorted and just is example. it seems each value of count will be printed instead of printing last amount of count

4

Are you creating this field? task_id => "%{name1}_%{name2}" would probably work.

5

yes task_id is value of field aggregat_id which is based on value of two other fields

6

indeed value of task_id will be as following:
c1_s1
c1_s2
c2_s1

7

based on output, it is counting the events which satisfy task_id but it is expected to print the last value of count which satisfy the task_id but it will print every value of count(count=1,count=2,count=3,.......)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.