Filebeat 1.1.0: Multiline Patterns

Elastic Stack Beats
1

Hi,
Does filebeat have any definitions for common patterns that need to be matched in log lines? Could the logstash grok patterns be reused? Is there a way I could make a set of pattern definitions available to filebeat?

Regards,
David

2

Grok support in beats is something that pops up quite frequently. See https://discuss.elastic.co/search?q=grok%20category%3A42

I assume the problem in your case is that the regexp get too complex? Currently the best way would be to extend the docs of filebeat with these patterns so others can just copy /paste it. A good place would be probably here: https://github.com/elastic/beats/blob/master/libbeat/docs/regexp.asciidoc

3

To be honest, I haven't really encountered a good use-case one needs the complicated patterns (even if abstracted away by grok). Trick is to not look at content as is (no need to write a 'full' regular parser), but look for patterns/shape of content.

Disadvantage of 'overcomplicated' patterns (sure they are not, but for use case of merging lines often is), is increased processing time by regex engine.

We're watching the forum and trying to collect use cases for documentation purposes. Any use-case you want to share?

4

Well, I'm looking at cassandra style logs which have a format like this:

<Log Level> [<Component>] <Datestamp> <Timestamp> <Message>
5

well, that's only half of the story, how do multline logs look like exactly? Assuming multiline is just stack traces starting with spaces a pattern like '^[[:space:]]+ might do the trick.

6

When I uncomment this bit of the config no log lines get processed at all.

7

Sorry, I don't understand what you're talking about.

8

The multiline config is commented out. It is commented out because when it is active all processing stops because of the error reported earlier.

9

config file is yaml and pretty sensitive to indentation and so on. Can you please share your filebeat.yml file so I can have a look?

If it continues to fail due to error reported earlier, does it mean you didn't change the regex to '^[[:space:]]'?

10

Ok, here's my prospector config:

filebeat:
    prospectors:
    -   document_type: cassandra
        input_type: log
        paths:
          - /var/log/cassandra/*.log
          - /var/log/cassandra/audit/*.log
        scan_frequency: 5s
        ignore_older: 168h
        multiline:
          pattern: '^\s'
          match: after
11

Seems like the indentation has been lost...

12

Changing the config to use the modified regex format you described seems to work...thank you:)