when i used three aggregate filter to aggregate first based on IW and then based on IT and at the end based on the AC, the output is multiple events where it is expected to be just one
You have in your example 10 log lines which are processed as 10 separate events in Elasticsearch. But you expect the result to be 2 log events before you index them to Elasticsearch? And also the order if where you put the fields are on a different place then where the event original sees it.
Unless I totally miss the point here, I do not see how you could solve that in Logstash. If you can guaranty the order the events are coming in you could do something with the multiline codec but I highly doubt that is feasible in a production environment.
You have in your example 10 log lines which are processed as 10 separate events in Elasticsearch. But you expect the result to be 2 log events before you index them to Elasticsearch?
hi, in this example yes, number of lines may be different.
And also the order if where you put the fields are on a different place then where the event original sees it
Could you please explain more. i cannot get it. if you mean the place of fields as AC, p1, p2 ,...; their location is fixed and i can extract these fields using grok filter
If you can guaranty the order the events are coming in you could do something with the multiline codec but I highly doubt that is feasible in a production environment.
actually the order of lines may differ. the point is that the lines can be linked based on the multi parameter. first based on the IW and IT parameter and then based on the AC parameter
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.