# Filebeat custom module docker multiline handling

**URL:** <https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 4, 2018, 10:53am UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034 "2018-10-04T10:53:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mvasilenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvasilenko/32/32886_2.png) [@mvasilenko](https://discuss.elastic.co/u/mvasilenko)\
**Post date:** [October 4, 2018, 10:53am UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034/1 "2018-10-04T10:53:32Z")

</div>

Hello,

I wrote small custom filebeat module for our web app logs parsing,  
but unable to catch multiline messages with exceptions,  
we are using filebeat at host, app is running in docker containers,

the question is how can i match multiline correctly,  
using grok patterns in my module pipeline.json,  
or in filebeat.yml?

example lines from log, what i want, is to add line with `'str' object has no attribute 'get'` to previous line

```auto
[2018-10-04 10:50:41 +0000] [ERROR] validation_plugin::validation_plugin.py::validate_data():L65:Exception at Validation process:
'str' object has no attribute 'get'
[2018-10-04 10:50:41 +0000] 10.0.1.101 (HTTP/1.1 200) GET / : Retned 34976 bytes in 60 msecs to "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"

```

my config:

```auto
filebeat.autodiscover:
  providers:
   - type: docker
     templates:
       - condition.or:
           - contains.docker.container.name: "backend"
         config:
           - module: custommodule
             backend:
               prospector:
                 type: docker
                 containers.stream: stderr
                 multiline.pattern: '^\['
                 multiline.negate: true
                 multiline.match: after
                 containers.ids:
                   - "${data.docker.container.id}"
         error:
               prospector:
                 type: docker
                 containers.stream: stderr
                 containers.ids:
                   - "${data.docker.container.id}"

```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 4, 2018, 11:58am UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034/2 "2018-10-04T11:58:18Z")

</div>

Your current solution is fine, it should be working. However, multiline configuration of Filebeat modules are in the files under `{module}/{fileset}/config/{fileset}.yml`. An example is the configuration of kafka/log fileset: [https://github.com/elastic/beats/blob/master/filebeat/module/kafka/log/config/log.yml#L7](https://github.com/elastic/beats/blob/master/filebeat/module/kafka/log/config/log.yml#L7)

---

<div class="post-metadata">

**Author:** ![mvasilenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvasilenko/32/32886_2.png) [@mvasilenko](https://discuss.elastic.co/u/mvasilenko)\
**Post date:** [October 5, 2018, 6:33am UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034/3 "2018-10-05T06:33:43Z")

</div>

Hi, Noémi, thank you for the answer,

but unfortunately, this solution doesn't seems to work for me,  
it still do not concatenate lines, at the same time,  
multiline settings works fine without module context

`/usr/share/filebeat/module/custommodule/backend/config/backend.yml`

```auto
type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
  pattern: '^\['
  negate: true
  match: after

```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 5, 2018, 9:55am UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034/4 "2018-10-05T09:55:29Z")

</div>

Does it work without autodiscover and docker prospector? Is Filebeat able to aggregate lines when reading from a local file?

---

<div class="post-metadata">

**Author:** ![mvasilenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvasilenko/32/32886_2.png) [@mvasilenko](https://discuss.elastic.co/u/mvasilenko)\
**Post date:** [October 5, 2018, 2:25pm UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034/5 "2018-10-05T14:25:29Z")

</div>

not sure, i'm using it only with autodiscover, because i need to handle dynamic container names

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2018, 2:25pm UTC](https://discuss.elastic.co/t/filebeat-custom-module-docker-multiline-handling/151034/6 "2018-11-02T14:25:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.