Start/Stop pattern for multiline

Elastic Stack Beats
1

I can't figure out how to use the filebeat multiline functionality to aggregate output with a start & stop pattern:

2016-03-29T09:45:01 process-A starting
2016-03-29T09:45:02 process-A foo
2016-03-29T09:45:03 process-A bar
2016-03-29T09:45:04 process-A finished
2016-03-29T09:45:05 process-B baz

Is that possible with current filebeat (1.2.0)? If not, is that on the road map? Thanks!

2

start/stop is not supported yet, but has been originally considered. Check ticket #461.

In your sample, is 'process-B' a single line event? If log file only consists of multiline events, it's possible to workaround this limitation by clever configs.

E.g. if process-A is always multiline but process-B is not one can filter for process-A.

3

It's syslog, where most lines/msgs stand on their own. process-A is the output from a cron job where individual lines as elasticsearch documents don't make sense (boot msgs might be another example). And a 3rd case is a log line that bursts and doesn't have any interesting per-msg data:

2016-03-29T09:45:06 process-C some generic error msg
...
2016-03-29T09:45:08 process-C some generic error msg

Would be nice to be able to roll those up into a single document and avoid "spamming" the elasticsearch index.

4

sounds like a mumbo-jumbo of everything. One can try to get some multiline support by using the regex OR-operator |, but then match order (before/after) and negate parameter must be same for all multiline patterns.

In general this use-case is not fully supported by multiline yet. Having a look at the actual logs, maybe (only maybe) I could come up with some regex pattern.

One problem with mutliline and all syslog output into one file might be interleaving multiline-events. Even if we were to support start/end patterns, you might still run into more subtle problems. Log routing as supported by some syslog daemons might help untangling the logs before forwarding logs via some shipper.

5

You may want to look at the aggregate filter in LS to handle this instead.