Filebeat - Multiline & Exclude Lines Combination

Elastic Stack Beats
1

Hi there,

We're currently using Filebeat to ship a log file into Logstash where fields are transformed for searching on in Elasticsearch however I've come into an issue I'm hoping someone here can help with.

So the idea is to store the contents of the log entry in 1 message field in Elasticsearch, here is a sample log file:

01-02-2018 11:00:01 GMT - NOTICE - Cron started.
01-02-2018 11:00:01 GMT - INFO - Processing files from the HTTP server.
01-02-2018 11:00:01 GMT - INFO - About to download and process these files: Array
(
    [0] => https://www.dummyserver.com/files/file.zip
)
01-02-2018 11:00:01 GMT - DEBUG - Current memory usage pre-loop: 10 mb.
01-02-2018 11:00:06 GMT - NOTICE - Total time taken: 00:05 (M:S)
01-02-2018 11:00:06 GMT - NOTICE - Cron finished with return status 0.

We're using the below filebeat config to store the above log into one message field:
- type: log
enabled: true
paths:
- /var/log/sample-log.log
exclude_lines: ['.INFO.']
fields:
tag: sample-log
multiline.pattern: "(?s)Cron started.*Cron end."
multiline.negate: true
multiline.match: after

The issue is, we now want to extract log lines that include INFO and DEBUG so only those lines with NOTICE are sent onto logstash but obviously as this multiline is putting the whole log example above into one line it appears as though filebeat can't read the lines with INFO in for example.

I was wondering if there was any option or method for achieving our desired results at the moment?

Many thanks.

2

You are using exclude_lines which excludes line which match the regex you provided. As your regex matches every line containing "INFO", that's why it seems like Filebeat cannot read lines with INFO.

You could use include_lines prospector options to select only messages which include NOTICE.

onclude_lines: ['.NOTICE.']

Thus, only NOTICE level messages are forwarded to Logstash.

3

Hi @kvch,

As the initial line has a loglevel of NOTICE, this is attaching everything into one log so the whole log entry is flagged as loglevel type NOTICE, as a result the solution you've provided doesn't work unfortunately.

Do you have any other ideas?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.