Filebeat Multiline multiple patterns

Elastic Stack Beats
1

I have a log file that has multiple, multi line events.

For example I have:

[24-Mar-2016 03:27:31] PHP Stack trace:
[24-Mar-2016 03:27:31] PHP   1. {main}() /home/app/www/myapp.com/feed/feed.php:0
[24-Mar-2016 03:27:31] PHP   2. require_once() /home/app/www/myapp.com/feed/feed.php:6
[24-Mar-2016 03:27:31] PHP   3. trigger_error() /home/app/php/db_mysql_connect_legacy.php:9

And

[24-Mar-2016 09:59:06] EmailQueue::process - Failed to send email to '' from 'user@myapp.com' (lid: vpa_s).  Errors: Array
(
    [0] => Missing destination email
)

Then everything else starts with

[24-Mar-2016 09:59:06]

How does the multiline handle situations like that? Whats the best way to match against multiple patterns in the same file?

2

multiple multiline events of differrent kinds are not well supported yet. But the regex library has an OR operation '|' that might be helpful in your case:

You can play with content and regex pattern yourself. Just press the run button and check output. All lines starting with 'false' will start a new multiline event.

I used the pattern pattern: '^\[.*\] PHP[[:space:]]+[[:digit:]]|^[^\[]'. The first part ^\[.*\] PHP[[:space:]]+[[:digit:]] captures traces (thanks to having a timestamp and having content PHP <digit>) and the second part (^[^\[]) collects all lines not beginning with '['.

3

Ahh almost prefect, that makes sense, I was using the wrong regex patterns. Turns out that \W and \d etc are not supported, that info was a little buried i the docs. but Im only getting the first line of the stack trace. Looks like I'll need to tweak the regex a bit, or use a different method on the multiline options.

4

Can you please post your config? I have this problem too.

5

I'm currently running filebeat as a windows service. I haven't tried this yet, but I wonder if I defined multiple windows services, which each run an instance of filebeat with it's own .yml file. That way, each yml file could direct filebeat to a specific log with a specific multiline pattern for it.