Filebeat Multiline Docker Log

Elastic Stack Beats
1 
filebeat.inputs:
  - type: log
    paths:
      - "/var/lib/docker/containers/*/*.log"
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: log
    processors:
    - rename:
        fields:
        - from: "log"
          to: "log.message"
        when:
          not:
            has_fields: ['log.flags']
    - add_docker_metadata: ~
    fields:
      application_name: <%= node.chef_environment %>
      log_type: "container_logs"
    multiline.pattern: (\d{4})-(\d{2})-(\d{2})
    multiline.negate: true
    multiline.match: after

Logs:

{"log":"2019-02-04 20:15:25.740 ERROR [reporting,,,] --- [enerContainer-1] c.o.r.s.impl.ReportExportServiceImpl     : createNotificationTask:: exportJobStatus: {} attachmentRequestStatus:{}\n","stream":"stdout","time":"2019-02-04T20:15:25.741261208Z"}
{"log":"\u0009... 58 common frames omitted\n","stream":"stdout","time":"2019-02-04T20:15:25.741612224Z"}
{"log":"\n","stream":"stdout","time":"2019-02-04T20:15:25.741615424Z"}
{"log":"2019-02-04 20:15:34.339  INFO [reporting,23452345,234523453245,false] --- [TaskScheduler-1] o.r.a.ReportAggregationPipelineProcessor : Failed to acquire lock for DATASUBJECTS\n","stream":"stdout","time":"2019-02-04T20:15:34.339299536Z"}

The logs are flowing in fine when it not multiline (The fourth line) but for the first three lines there is a just a single event I can see in kibana with log.flags multiline and no log or message field. This config still works with filebeat 6.4.x version but does not work with changes in 6.5.x version of the filebeat.

2

I think this might be a one off issue because I am using the json plugin as well as with multiline.
When I add json.overwrite_keys: true I start getting the logs again but then the newly added flags in the filebeat for multiline is missing

3

Did you try to use the docker input to read the log? It setups a few things by default.

4

Maybe bacause your multiline.pattern should be like this:

multiline.pattern: ^(\d{4})-(\d{2})-(\d{2})

That's the only difference I can see with what we use here for docker.

5

@TimTim I dont think that is the issue. What is your version of filebeat. As I have mentioned above things works perfectly fine for versions < 6.5.0

6

@pierhugues will try that. But I think the issue is log attribute is getting overriden by multiline filter. json adds a log field which has the log and so that the multiline now and it overwrites it causing to lose the log message.

7

My bad, missed that bit.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.