How to aggregate two events from log lines with no common pattern

(Jose) #1

Hi

I am reading the documentation on the Aggregate Filter, but all examples assume there is something common between the lines (events) to be aggregated. Usually, that common entity is also used as task_id.
But my case is like this:

...
something happened on hostname a.b.c because this and that
...
the action was done by user joe because blah blah
...

From those log lines, I need LogStash to create a single document like

{
    "hostname" => "a.b.c"
    "user" => "joe"
}

Is there an easy way to do it?
Any tip is more than welcome.

Cheers,
Jose

#2

If your log always has pairs of those two lines you could do something like this.

You would need to grok the fields you want and for the hostname stash that in a class variable in the first ruby filter.

(Jose) #3

thanks a lot !!!
I am going to have a look into that right now.

(Jose) #4

follow up question. Is it possible to add a time out? I am not parsing only those 2 lines I mentioned, but 5 lines. The 5th one, for the current action, may not be yet in the log file. So, I would like to send to the output the data I already have for the first 4 variables if the 5th one is not ready in less than 30 seconds, for example. Is that possible?

#5

Well, it's ruby, so it can do pretty much anything. However, I wouldn't know how to do it.

(Jose) #6

:slight_smile:
I am investigating one possible solution: chaining 2 pipelines. In the first one, I sent to the output (tcp) both the json with the first 4 fields and the json with 5 fields, when possible. I read all of that in the second pipeline (input tcp). I hope I can figure out now how to use the aggregate filter in the second pipeline to spot when I have both the partial and the complete event, and drop the partial one. Let's see...

(system) closed #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.