Combine request and response based on field in log - ELK

We have ELK setup with filebeat, logstash and elasticsearch and kibana. I need aggregate request and response at the logstash.

I have configured pipeline configuration as below. now log aggregation working without any issue if i use single worker for pipeline. if i use multiple workers aggregation is not happening. any solution to use multiple workers and log aggregation?

 if [transaction] == "request" { 

       aggregate {
                        task_id => "%{id}"
                        code => "
                        map['method'] = event.get('method')
                        map['request'] = event.get('request')
                        map['user'] = event.get('user')
                        map['application'] = event.get('application')"
                        map_action => "create"
         }
                    drop {}#drop the request before persisting, to save indexing space in elasticsearch server
  }
  if [message] =~ "TRANSACTION:response" {

         aggregate {
                    task_id => "%{id}"
                    code => "
                    event.set('method', map['method'])
                    event.set('response', map['response'])
                    event.set('user', map['user'])
                    event.set('application', map['application'])"
                    map_action => "update"

         }
}
1 Like

No. You can only use one worker with an aggregate filter.

1 Like