How can merge logs based on the multi parameters

[sahere37 (LR)]

hi all
I have a question, how can merge multi lines of logs based on the multi parameters into one event?. for example, log is as following:

IT=1  AC=12345 p1=12345
IW=1 AC=12345 A1=check1 
IW=1 E1=0
IW=2 AC=12345 A1=check2
IW=2 E1=4
IT=1 p2=ok
IT=2  AC=7894 p1=245
IW=4 AC=7894 A1=get 
IW=4 E1=2
IT=2 p2=failed

now the expected result is as following:

   IT=1 AC=12345 p1=12345 IW={1,2} A1={check1,check2} E1={0,4} p2=ok
   IT=2 AC=7894 p1=245 IW=4 A1=get E1=2 p2=failed

actually i want to aggregate events first based on the IW parameter:

IW=1 AC=12345 A1=check1 E1=0
IW=2 AC=12345 A1=check2 E1=4
IW=4 AC=7894 A1=get  E1=2

and then aggregate based on the IT parameter

IT=1  AC=12345 p1=12345 p2=ok
IT=2  AC=7894 p1=245 p2=failed

and at the end aggregate based on the AC parameter in one event

  AC=12345  IT=1  p1=12345 IW={1,2} A1={check1,check2} E1={0,4} p2=ok
  AC=7894   IT=2  p1=245 IW=4 A1=get E1=2 p2=failed

any advise will be so appreciated

[sahere37 (LR)]

when i used three aggregate filter to aggregate first based on IW and then based on IT and at the end based on the AC, the output is multiple events where it is expected to be just one

[pjanzen (Paul Janzen)]

Hi,

You have in your example 10 log lines which are processed as 10 separate events in Elasticsearch. But you expect the result to be 2 log events before you index them to Elasticsearch? And also the order if where you put the fields are on a different place then where the event original sees it.

Unless I totally miss the point here, I do not see how you could solve that in Logstash. If you can guaranty the order the events are coming in you could do something with the multiline codec but I highly doubt that is feasible in a production environment.

[sahere37 (LR)]

You have in your example 10 log lines which are processed as 10 separate events in Elasticsearch. But you expect the result to be 2 log events before you index them to Elasticsearch?
hi, in this example yes, number of lines may be different.

And also the order if where you put the fields are on a different place then where the event original sees it

Could you please explain more. i cannot get it. if you mean the place of fields as AC, p1, p2 ,...; their location is fixed and i can extract these fields using grok filter

If you can guaranty the order the events are coming in you could do something with the multiline codec but I highly doubt that is feasible in a production environment.

actually the order of lines may differ. the point is that the lines can be linked based on the multi parameter. first based on the IW and IT parameter and then based on the AC parameter

[sahere37 (LR)]

any update and solution? any advise will be so appreciated