What does filebeat's multiline tool match

Elastic Stack Beats
1

I am using filebeat to read Docker logs and feed them to logstash. I have some logs that are being split into separate events. This happens with stack traces, or just any logs with a new line in them.

Here is my filebeat.yml file.

    filebeat.autodiscover:
        providers:
            - type: docker
                templates:
                    - config:
                        - type: container
                            paths:
                                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
    logging.metrics.enabled: false
    output.logstash:
        hosts:
            - logstash:5044

I have a couple questions about this. First off, if I look in those .log files, they are in the following format

    {"log":"Message is in here\n", "stream":"stderr", "time":"<timestamp here>"}
    {"log":"\u0009this line should be added to the previous log\n", "stream":"stderr", "time":"<timestamp here>"}

In the multiline.pattern field, is it looking to match the log part only, or the entire line starting with the {? Basically it comes down to should I use

multiline.pattern: '^{"log":"\t'

or

multiline.pattern: '^\t'

In addition, will this tool even work because i just want to take the log part and append it to the previous log message?

2

Hi!

Since you want to handle json logs I would suggest you playing around with the json related features that Filebeat provides:

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json
https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html

3

Hi so this is now my filebeat.yml. Does this look roughly right to you?

filebeat.autodiscover:
    providers:
        - type: docker
            templates:
                - config:
                    - type: container
                        paths:
                            - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
            json.message_key: log
            multiline.pattern: ^\t
            multiline.negate: false
            multiline.match: after
logging.metrics.enabled: false
output.logstash:
    hosts:
        - logstash:5044
4

I found the solution for my case. This is now my filebeat.yml file

filebeat.autodiscover:
        providers:
            - type: docker
                templates:
                    - config:
                        - type: container
                            paths:
                                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
                        multiline.pattern: "^\t"
                        multiline.negate: false
                        multiline.match: after
    logging.metrics.enabled: false
    output.logstash:
        hosts:
            - logstash:5044

I had the three multiline lines at the wrong "level". Originally they were right between the - type: docker and templates: lines.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.