Fixing Multiline Logs in Filebeat

[kevin5617]

I know that filebeat has a multiline tool to handle multiline logs. However, the way my log structure is setup I do not think those solutions work exactly the same way. I am using the ELK stack with docker, and for filebeat, I am using the "autodiscover" tool to read docker logs.

When directing my filebeat output to a file, the logs are in the form of

`{"@timestamp":"2020-03-19T18:42:33.589Z",........."message":"INFO: stuff here"....}`

So I think filebeat will look to match a pattern for that entire log, not just the message field. So if I were to try to use multiline, would I have to do something like

    multiline.pattern: '^.*\"message\":\"<some_pattern_here>'
    multiline.negate: true
    multiline.match: after

Also, where would I put this inside the filebeat config. Would I put it one level under the autodiscover, like

    filebeat.autodiscover:
        multiline.pattern: '^.*\"message\":\"<some_pattern_here>'
        multiline.negate: true
        multiline.match: after
        providers:
        ....

I also don't just want to move the whole log that doesn't match the matter, I want to move the message field of that log and move it to the end of the message field of the previous log.

[shaunak (Shaunak Kashyap)]

Perhaps I'm misunderstanding but it sounds like the multiline messages are encoded within the message field, right? If so, wouldn't there be \n characters in there already? If so, do you even need the multiline.* settings?

It might be helpful if you could copy-paste a small chunk (few lines) from your Docker logs. If it doesn't fit in here, feel free to use https://gist.github.com/ or https://pastebin.com/ and post the link here.

Shaunak

[kevin5617]

So i'm not totally sure what the multiline plugin is "looking at" to use the pattern. I'm using the autodiscover tool to get docker logs. so if I look at the locations where the docker logs are from, and cat the <docker_id>-json.log file, the lines are in the form

{"log":"INFO: message here\n","stream":"stderr","time":"<timestamp>"}

and then for the "indented" lines which I want to attach to the previous message, they look like

{"log":"\u0009this should be part of the previous log\n","stream":"stderr","time":"<timestamp>"}

let me know if this answers your question

[kevin5617]

I found the solution for my case. This is now my filebeat.yml file

filebeat.autodiscover:
        providers:
            - type: docker
                templates:
                    - config:
                        - type: container
                            paths:
                                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
                        multiline.pattern: "^\t"
                        multiline.negate: false
                        multiline.match: after
    logging.metrics.enabled: false
    output.logstash:
        hosts:
            - logstash:5044

I had the three multiline lines at the wrong "level". Originally they were right between the - type: docker and templates: lines.