Start/Stop pattern for multiline

(Erik Stephens) #1

I can't figure out how to use the filebeat multiline functionality to aggregate output with a start & stop pattern:

2016-03-29T09:45:01 process-A starting
2016-03-29T09:45:02 process-A foo
2016-03-29T09:45:03 process-A bar
2016-03-29T09:45:04 process-A finished
2016-03-29T09:45:05 process-B baz

Is that possible with current filebeat (1.2.0)? If not, is that on the road map? Thanks!

(Steffen Siering) #2

start/stop is not supported yet, but has been originally considered. Check ticket #461.

In your sample, is 'process-B' a single line event? If log file only consists of multiline events, it's possible to workaround this limitation by clever configs.

E.g. if process-A is always multiline but process-B is not one can filter for process-A.

(Erik Stephens) #3

It's syslog, where most lines/msgs stand on their own. process-A is the output from a cron job where individual lines as elasticsearch documents don't make sense (boot msgs might be another example). And a 3rd case is a log line that bursts and doesn't have any interesting per-msg data:

2016-03-29T09:45:06 process-C some generic error msg
2016-03-29T09:45:08 process-C some generic error msg

Would be nice to be able to roll those up into a single document and avoid "spamming" the elasticsearch index.

(Steffen Siering) #4

sounds like a mumbo-jumbo of everything. One can try to get some multiline support by using the regex OR-operator |, but then match order (before/after) and negate parameter must be same for all multiline patterns.

In general this use-case is not fully supported by multiline yet. Having a look at the actual logs, maybe (only maybe) I could come up with some regex pattern.

One problem with mutliline and all syslog output into one file might be interleaving multiline-events. Even if we were to support start/end patterns, you might still run into more subtle problems. Log routing as supported by some syslog daemons might help untangling the logs before forwarding logs via some shipper.

(Mark Walkom) #5

You may want to look at the aggregate filter in LS to handle this instead.

(system) #6