Need to use multiple events and build single aggregated event based on session id to send to ES
Here is how data looks like:
session open for user abc (session id :1)
session open for user bbc (session id :2)
....some operation by user abc (session id :1)
....some openration by user bbc (session id :2)
session closed for user abc (session id :1)
...some operation by user bbc (session id :2)
session closed for user bbc (session id :2)
Am using aggregate filter. Also specified event.cancel() for non final events.
Know the start and end events . Have set pipeline workers as 1
However, all the events are sent to ES. Checked this post ([AGGREGATE] - aggregate is working but single rows are sent to elastic too) and looks like it should work though.
Noticed in one of the other posts (Aggregate - concatenate events), it suggests to use multiline in filebeat as well (along with aggregate plugin).
Is FB multiline setting really needed if i have to avoid all events to be sent to ES ??