# Filebeat filestream with pipeline and multiline

**URL:** <https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 1, 2023, 4:10pm UTC](https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015 "2023-06-01T16:10:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![das](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/das/32/121758_2.png) [@das](https://discuss.elastic.co/u/das)\
**Post date:** [June 1, 2023, 4:10pm UTC](https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015/1 "2023-06-01T16:10:58Z")

</div>

I have a log format I cannot change that leads into multiline messages. I have a Ingest Pipeline set up in Kibana that works just fine on sample records. My problem is that My multiline parser seems to be ignored (at least in error) when I attempt to configure both.

Here's what my filebeat configuration looks like:

```auto
filebeat.inputs:
  - type: filestream
    enabled: true
    id: my_env
    index: my_env
    pipeline: mycustompipeline
    paths:
      - /opt/log-mounts/my_env/**/output.log
    fields:
      environment: my_env
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
      negate: true
      match: after

```

Log messages are output based on this configuration from Log4J2 in a Java project (again, I can't just add the JSON appender as it's a COTS app that won't allow it):

```auto
<Pattern>%d [%20.20t] [%10.10X{key1}] [%20.20X{key2}] [%20.20X{key3}] (%30.30c{3}) %-5p %X{key4} %X{key5} %X{key6} - %m%n</Pattern>

```

What do I need to do to have the multiline parse correctly **AND** run before the pipeline?

---

<div class="post-metadata">

**Author:** ![das](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/das/32/121758_2.png) [@das](https://discuss.elastic.co/u/das)\
**Post date:** [June 1, 2023, 4:35pm UTC](https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015/2 "2023-06-01T16:35:13Z")

</div>

I got most stuff working:

```auto
filebeat.inputs:
  - type: filestream
    enabled: true
    id: my_env
    index: my_env
    paths:
      - /opt/log-mounts/my_env/**/output.log
    fields:
      environment: my_env

    multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
    multiline.negate: true
    mutiline.match: after

    pipeline: mycustompipeline

```

I just need to get it to include blank lines in multieline parsing.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 2, 2023, 4:19am UTC](https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015/3 "2023-06-02T04:19:57Z")

</div>

Hi @das What version filebeat, that is not the correct syntax for multiline using ` - type: filestream`

That is the old / previous syntax that goes with the deprectation ` - type: log` input

> [@das](#):
>
> ```auto
> multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
> multiline.negate: true
> mutiline.match: after
> 
> ```

See [here](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)

Should look like

```auto
parsers:
- multiline:
    type: pattern
    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
    negate: true
    match: after

```

---

<div class="post-metadata">

**Author:** ![das](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/das/32/121758_2.png) [@das](https://discuss.elastic.co/u/das)\
**Post date:** [June 2, 2023, 1:12pm UTC](https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015/4 "2023-06-02T13:12:44Z")

</div>

I was able to fix this! What you have there is correct!  
I got backed up and forgot to post the final solution I came up with:

```auto
filebeat.inputs:
  - type: filestream
    enabled: true
    id: my_env
    index: my_env
    paths:
      - /opt/log-mounts/my_env/**/output.log
    fields:
      environment: my_env
    parsers:
      - multiline:
          pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
          negate: true
          match: after
    pipeline: mycustompipeline

```

Once I got everything squared away and realizing the documentation I was reading and even ChatGPT were not the right version, this all flowed together really fast. I also was able to verify that the multiline parser is picking up blank lines appropriate as well.

My lesson was most definitely:  
**MAKE SURE YOU ARE ON THE RIGHT DOCUMENTATION VERSION**

There's some weirdness with the app where it seems to ignore this at random, but that's not an issue for here:

```auto
%X{key4} %X{key5} %X{key6}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2023, 3:12pm UTC](https://discuss.elastic.co/t/filebeat-filestream-with-pipeline-and-multiline/335015/5 "2023-06-30T15:12:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.