# Filebeat - Multiline & Exclude Lines Combination

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-exclude-lines-combination/120193>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 16, 2018, 2:17pm UTC](https://discuss.elastic.co/t/filebeat-multiline-exclude-lines-combination/120193 "2018-02-16T14:17:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amgryptio](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@Amgryptio](https://discuss.elastic.co/u/Amgryptio)\
**Post date:** [February 16, 2018, 2:17pm UTC](https://discuss.elastic.co/t/filebeat-multiline-exclude-lines-combination/120193/1 "2018-02-16T14:17:50Z")

</div>

Hi there,

We're currently using Filebeat to ship a log file into Logstash where fields are transformed for searching on in Elasticsearch however I've come into an issue I'm hoping someone here can help with.

So the idea is to store the contents of the log entry in 1 message field in Elasticsearch, here is a sample log file:

```
01-02-2018 11:00:01 GMT - NOTICE - Cron started.
01-02-2018 11:00:01 GMT - INFO - Processing files from the HTTP server.
01-02-2018 11:00:01 GMT - INFO - About to download and process these files: Array
(
    [0] => https://www.dummyserver.com/files/file.zip
)
01-02-2018 11:00:01 GMT - DEBUG - Current memory usage pre-loop: 10 mb.
01-02-2018 11:00:06 GMT - NOTICE - Total time taken: 00:05 (M:S)
01-02-2018 11:00:06 GMT - NOTICE - Cron finished with return status 0.

```

We're using the below filebeat config to store the above log into one message field:  
- type: log  
enabled: true  
paths:  
- /var/log/sample-log.log  
exclude\_lines: ['._INFO._']  
fields:  
tag: sample-log  
multiline.pattern: "(?s)Cron started.\*Cron end."  
multiline.negate: true  
multiline.match: after

The issue is, we now want to extract log lines that include INFO and DEBUG so only those lines with NOTICE are sent onto logstash but obviously as this multiline is putting the whole log example above into one line it appears as though filebeat can't read the lines with INFO in for example.

I was wondering if there was any option or method for achieving our desired results at the moment?

Many thanks.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 16, 2018, 2:55pm UTC](https://discuss.elastic.co/t/filebeat-multiline-exclude-lines-combination/120193/2 "2018-02-16T14:55:16Z")

</div>

You are using `exclude_lines` which excludes line which match the regex you provided. As your regex matches every line containing "INFO", that's why it seems like Filebeat cannot read lines with INFO.

You could use `include_lines` prospector options to select only messages which include NOTICE.

```auto
onclude_lines: ['.NOTICE.']

```

Thus, only NOTICE level messages are forwarded to Logstash.

---

<div class="post-metadata">

**Author:** ![Amgryptio](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@Amgryptio](https://discuss.elastic.co/u/Amgryptio)\
**Post date:** [February 16, 2018, 4:15pm UTC](https://discuss.elastic.co/t/filebeat-multiline-exclude-lines-combination/120193/3 "2018-02-16T16:15:44Z")

</div>

Hi @kvch,

As the initial line has a loglevel of NOTICE, this is attaching everything into one log so the whole log entry is flagged as loglevel type NOTICE, as a result the solution you've provided doesn't work unfortunately.

Do you have any other ideas?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2018, 4:16pm UTC](https://discuss.elastic.co/t/filebeat-multiline-exclude-lines-combination/120193/4 "2018-03-16T16:16:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.