# Use beats to read multiline event excluding some lines in-between

**URL:** <https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 1, 2021, 7:22am UTC](https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990 "2021-04-01T07:22:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lukiovas](https://avatars.discourse-cdn.com/v4/letter/l/c5a1d2/32.png) [@lukiovas](https://discuss.elastic.co/u/lukiovas)\
**Post date:** [April 1, 2021, 7:22am UTC](https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990/1 "2021-04-01T07:22:50Z")

</div>

I have a log:

```auto
 server.name 2021-03-28 10:03:28.648 INFO ...
---------------------------
ID: 3974
Address: https://www...
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml
Headers: {Accept=[*/*], ...
Payload: <soap:Envelope...

```

I want filebeat to read this log as multiline event as so:

```auto
server.name 2021-03-28 10:03:28.648 INFO ...
Payload: <soap:Envelope...

```

---, ID, Address, Encoding, Http-Method, Content-Type, Headers should be excluded. I tried following config in filebeat.yml unsuccessfully :

```auto
- type: log
  enabled: true

  paths:
    - "/var/log/app_servers/liferay-ext.log"
    - "/var/log/app_servers/liferay-int.log"

  fields :

    log_type: "logserver_prod_vrklt_liferay"
  
  exclude_lines: ['---','ID:','Address:','Encoding:','Http-Method:','Content-Type:','Headers:']

  multiline.pattern: ^\server.name
  multiline.match: after
  multiline.flush.pattern: 'Payload: '

```

Any ideas on how to get my desired result?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 1, 2021, 10:27am UTC](https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990/2 "2021-04-01T10:27:52Z")

</div>

Hi @lukiovas, welcome to discuss 🙂

`exclude_lines` is executed after `multiline` groups the lines in a single field, so with the configuration you are trying I guess that lines are being grouped, but then they are excluded because they contain the pattens in `exclude_lines`.

I would suggest to use multiline to group all these lines, and then use [processors](https://www.elastic.co/guide/en/beats/filebeat/7.12/filtering-and-enhancing-data.html) to extract the relevant information. For example you could try to use [`dissect`](https://www.elastic.co/guide/en/beats/filebeat/7.12/dissect.html) to extract the relevant fields, and then [`drop_fields`](https://www.elastic.co/guide/en/beats/filebeat/7.12/drop-fields.html) to drop the original log message if you don't want to keep it.

---

<div class="post-metadata">

**Author:** ![lukiovas](https://avatars.discourse-cdn.com/v4/letter/l/c5a1d2/32.png) [@lukiovas](https://discuss.elastic.co/u/lukiovas)\
**Post date:** [April 2, 2021, 8:31am UTC](https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990/3 "2021-04-02T08:31:20Z")

</div>

@jsoriano , thank you for your advice.

Firslty I'm trying to drop all the events that are not multiline. The strict yml configuration syntax is giving me a hard time. Could you give me an idea of whats wrong with the following processor:

```auto
processors:
   - drop_event:
       when:
          not:
          equals:
             flags: "multiline"

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 6, 2021, 10:52am UTC](https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990/4 "2021-04-06T10:52:59Z")

</div>

Be careful with the indentation, `equals` should have one more indentation level than `not`. Also, `flags` seems to be an array, and I am not sure if this condition works with arrays, you can also try with `contains`.

So try this:

```auto
processors:
   - drop_event:
       when:
         not:
           equals:
             flags: "multiline"

```

Or, as they are all objects with single members, you can simplify the condition like this:

```auto
processors:
   - drop_event:
       when.not.equals.flags: "multiline"

```

Try with `contains` instead of `equals` if it doesn't work after fixing indentation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 4, 2021, 12:53pm UTC](https://discuss.elastic.co/t/use-beats-to-read-multiline-event-excluding-some-lines-in-between/268990/5 "2021-05-04T12:53:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.