Aggregate plugin sends all events to ES

Elastic Stack Logstash
1

Hi ,

Need to use multiple events and build single aggregated event based on session id to send to ES

Here is how data looks like:
session open for user abc (session id :1)
session open for user bbc (session id :2)
....some operation by user abc (session id :1)
....some openration by user bbc (session id :2)
session closed for user abc (session id :1)
...some operation by user bbc (session id :2)
session closed for user bbc (session id :2)

Am using aggregate filter. Also specified event.cancel() for non final events.
Know the start and end events . Have set pipeline workers as 1

However, all the events are sent to ES. Checked this post ([AGGREGATE] - aggregate is working but single rows are sent to elastic too) and looks like it should work though.
Noticed in one of the other posts (Aggregate - concatenate events), it suggests to use multiline in filebeat as well (along with aggregate plugin).
Is FB multiline setting really needed if i have to avoid all events to be sent to ES ??

2

What does your filter configuration look like?

3

It has grok plugin to fetch basic field to differentiate start, middle and end events
start aggregate plugin with event.cancel(), code, taskid, map settings
Another grok on basic field to fetch the specific operations and fill the map
Based on operations, multiple aggregate filters with event.cancel() in each and map_action "update"
Then a final aggregate filter for final event based on basic grok parsed field.
including below
map_action => "update"
end_of_task => true

Haven't set any timeout related fields as not needed as of now.

4

Solved the issue myself :slight_smile:
Moved all the grok plugins up and only the aggregate plugins down (although the condition statements were repeated )

5

I see the sequence of events is not maintained. How do we ensure that ?

6

Set pipeline.workers to 1 and (for now) set java_execution false.

7

Have already set pipeline worker as 1. Where do we set java_execution false ?

8

You can set it on the command line or in logstash.yml

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.