How to manage multiline events based on a random field

Elastic Stack Logstash
21

Concerning the pattern in the "if" condition, you have to know that pattern has just to be "contained" in field. Logstash doesn't search an exact match but a "contain" match.
So the beginning .* is not necessary.

Concerning events that are not gathered, as mentioned previously, if you have multiple logstash threads and multiple events occur at the same time for the same "task_id", then you can't be sure that behaviour is nice, because functionally, for each "task_id", you need that events are processed in the right order.

So, if you have multiple events that can occur at the same time for one task_id, the only way to have a fully effective behaviour is to have only one logstash thread (command line option : -w 1).

22

Ok for the regular expression, logical, it could have worked with only "PP tunnel..." and I didn't notice it.

In the other hand, I am wondering; the log lines, before their gathering, are grokked. Once the aggregate filter is applied, it is still possible to make searches on grokked fields on the separate events, but not on the aggregated ones. Do I have to grok the aggregated events with different grok fields even if those events are the same as the separate ones?

It makes sense actually that the previousely grokked fields are not used for the aggregated events, because when the events are aggregated, the duplicates remains, so if I make a search on a particular event, I have twice the same result, and that would be absurd.

23

To be clear :

  • typically, grok filter parse message field to create specific fields
  • typically, aggregate filter gather some fields from events 0 to n-1 to finally push them into event n (last event from one task)
  • fields created using grok can be used by aggregate filter
  • after that aggregate filter push aggregated fields into last event, you can use grok to process aggregated fields.
1 Like
24

Yes actually that's the thing. When I make a search using grok fields I've got results when the search matches the low level events (the none aggregated ones). But I have no results when the filter I type in the search bar should match the aggregated events, as if those events are not related at all, even if they're tied up. A search including only the expressions I am looking for (without using grok fields) works fine tough.