Filebeat Multiline Docker Log

(Shaurya Garg) #1
  - type: log
      - "/var/lib/docker/containers/*/*.log"
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: log
    - rename:
        - from: "log"
          to: "log.message"
            has_fields: ['log.flags']
    - add_docker_metadata: ~
      application_name: <%= node.chef_environment %>
      log_type: "container_logs"
    multiline.pattern: (\d{4})-(\d{2})-(\d{2})
    multiline.negate: true
    multiline.match: after


{"log":"2019-02-04 20:15:25.740 ERROR [reporting,,,] --- [enerContainer-1] c.o.r.s.impl.ReportExportServiceImpl     : createNotificationTask:: exportJobStatus: {} attachmentRequestStatus:{}\n","stream":"stdout","time":"2019-02-04T20:15:25.741261208Z"}
{"log":"\u0009... 58 common frames omitted\n","stream":"stdout","time":"2019-02-04T20:15:25.741612224Z"}
{"log":"2019-02-04 20:15:34.339  INFO [reporting,23452345,234523453245,false] --- [TaskScheduler-1] o.r.a.ReportAggregationPipelineProcessor : Failed to acquire lock for DATASUBJECTS\n","stream":"stdout","time":"2019-02-04T20:15:34.339299536Z"}

The logs are flowing in fine when it not multiline (The fourth line) but for the first three lines there is a just a single event I can see in kibana with log.flags multiline and no log or message field. This config still works with filebeat 6.4.x version but does not work with changes in 6.5.x version of the filebeat.

(Shaurya Garg) #2

I think this might be a one off issue because I am using the json plugin as well as with multiline.
When I add json.overwrite_keys: true I start getting the logs again but then the newly added flags in the filebeat for multiline is missing

(Pier-Hugues Pellerin) #3

Did you try to use the docker input to read the log? It setups a few things by default.

(Timothy Eichmann) #4

Maybe bacause your multiline.pattern should be like this:

multiline.pattern: ^(\d{4})-(\d{2})-(\d{2})

That's the only difference I can see with what we use here for docker.

(Shaurya Garg) #5

@TimTim I dont think that is the issue. What is your version of filebeat. As I have mentioned above things works perfectly fine for versions < 6.5.0

(Shaurya Garg) #6

@pierhugues will try that. But I think the issue is log attribute is getting overriden by multiline filter. json adds a log field which has the log and so that the multiline now and it overwrites it causing to lose the log message.

(Timothy Eichmann) #7

My bad, missed that bit.