Filebeat custom module docker multiline handling

Elastic Stack Beats
1

Hello,

I wrote small custom filebeat module for our web app logs parsing,
but unable to catch multiline messages with exceptions,
we are using filebeat at host, app is running in docker containers,

the question is how can i match multiline correctly,
using grok patterns in my module pipeline.json,
or in filebeat.yml?

example lines from log, what i want, is to add line with 'str' object has no attribute 'get' to previous line

[2018-10-04 10:50:41 +0000] [ERROR] validation_plugin::validation_plugin.py::validate_data():L65:Exception at Validation process:
'str' object has no attribute 'get'
[2018-10-04 10:50:41 +0000] 10.0.1.101 (HTTP/1.1 200) GET / : Retned 34976 bytes in 60 msecs to "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"

my config:

filebeat.autodiscover:
  providers:
   - type: docker
     templates:
       - condition.or:
           - contains.docker.container.name: "backend"
         config:
           - module: custommodule
             backend:
               prospector:
                 type: docker
                 containers.stream: stderr
                 multiline.pattern: '^\['
                 multiline.negate: true
                 multiline.match: after
                 containers.ids:
                   - "${data.docker.container.id}"
         error:
               prospector:
                 type: docker
                 containers.stream: stderr
                 containers.ids:
                   - "${data.docker.container.id}"
2

Your current solution is fine, it should be working. However, multiline configuration of Filebeat modules are in the files under {module}/{fileset}/config/{fileset}.yml. An example is the configuration of kafka/log fileset: https://github.com/elastic/beats/blob/master/filebeat/module/kafka/log/config/log.yml#L7

3

Hi, Noémi, thank you for the answer,

but unfortunately, this solution doesn't seems to work for me,
it still do not concatenate lines, at the same time,
multiline settings works fine without module context

/usr/share/filebeat/module/custommodule/backend/config/backend.yml

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
  pattern: '^\['
  negate: true
  match: after
4

Does it work without autodiscover and docker prospector? Is Filebeat able to aggregate lines when reading from a local file?

5

not sure, i'm using it only with autodiscover, because i need to handle dynamic container names

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.