# Filebeat Multiline Docker Log

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 5, 2019, 3:37pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153 "2019-02-05T15:37:20Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![shauryagarg2006](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@shauryagarg2006](https://discuss.elastic.co/u/shauryagarg2006)\
**Post date:** [February 5, 2019, 3:37pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/1 "2019-02-05T15:37:21Z")

</div>

```auto
filebeat.inputs:
  - type: log
    paths:
      - "/var/lib/docker/containers/*/*.log"
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: log
    processors:
    - rename:
        fields:
        - from: "log"
          to: "log.message"
        when:
          not:
            has_fields: ['log.flags']
    - add_docker_metadata: ~
    fields:
      application_name: <%= node.chef_environment %>
      log_type: "container_logs"
    multiline.pattern: (\d{4})-(\d{2})-(\d{2})
    multiline.negate: true
    multiline.match: after

```

Logs:

```auto
{"log":"2019-02-04 20:15:25.740 ERROR [reporting,,,] --- [enerContainer-1] c.o.r.s.impl.ReportExportServiceImpl : createNotificationTask:: exportJobStatus: {} attachmentRequestStatus:{}\n","stream":"stdout","time":"2019-02-04T20:15:25.741261208Z"}
{"log":"\u0009... 58 common frames omitted\n","stream":"stdout","time":"2019-02-04T20:15:25.741612224Z"}
{"log":"\n","stream":"stdout","time":"2019-02-04T20:15:25.741615424Z"}
{"log":"2019-02-04 20:15:34.339 INFO [reporting,23452345,234523453245,false] --- [TaskScheduler-1] o.r.a.ReportAggregationPipelineProcessor : Failed to acquire lock for DATASUBJECTS\n","stream":"stdout","time":"2019-02-04T20:15:34.339299536Z"}

```

The logs are flowing in fine when it not multiline (The fourth line) but for the first three lines there is a just a single event I can see in kibana with log.flags multiline and no log or message field. This config still works with filebeat 6.4.x version but does not work with changes in 6.5.x version of the filebeat.

---

<div class="post-metadata">

**Author:** ![shauryagarg2006](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@shauryagarg2006](https://discuss.elastic.co/u/shauryagarg2006)\
**Post date:** [February 5, 2019, 4:30pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/2 "2019-02-05T16:30:13Z")

</div>

I think this might be a one off issue because I am using the json plugin as well as with multiline.  
When I add `json.overwrite_keys: true` I start getting the logs again but then the newly added flags in the filebeat for multiline is missing

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [February 7, 2019, 1:36pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/3 "2019-02-07T13:36:06Z")

</div>

Did you try to use the [docker input](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-docker.html) to read the log? It setups a few things by default.

---

<div class="post-metadata">

**Author:** ![TimTim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timtim/32/5295_2.png) [@TimTim](https://discuss.elastic.co/u/TimTim)\
**Post date:** [February 7, 2019, 2:27pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/4 "2019-02-07T14:27:38Z")

</div>

Maybe bacause your multiline.pattern should be like this:

```
multiline.pattern: ^(\d{4})-(\d{2})-(\d{2})

```

That's the only difference I can see with what we use here for docker.

---

<div class="post-metadata">

**Author:** ![shauryagarg2006](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@shauryagarg2006](https://discuss.elastic.co/u/shauryagarg2006)\
**Post date:** [February 7, 2019, 3:35pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/5 "2019-02-07T15:35:18Z")

</div>

@TimTim I dont think that is the issue. What is your version of filebeat. As I have mentioned above things works perfectly fine for versions \< 6.5.0

---

<div class="post-metadata">

**Author:** ![shauryagarg2006](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@shauryagarg2006](https://discuss.elastic.co/u/shauryagarg2006)\
**Post date:** [February 7, 2019, 3:36pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/6 "2019-02-07T15:36:45Z")

</div>

@pierhugues will try that. But I think the issue is log attribute is getting overriden by multiline filter. json adds a log field which has the log and so that the multiline now and it overwrites it causing to lose the log message.

---

<div class="post-metadata">

**Author:** ![TimTim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timtim/32/5295_2.png) [@TimTim](https://discuss.elastic.co/u/TimTim)\
**Post date:** [February 7, 2019, 3:48pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/7 "2019-02-07T15:48:35Z")

</div>

My bad, missed that bit.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2019, 3:48pm UTC](https://discuss.elastic.co/t/filebeat-multiline-docker-log/167153/8 "2019-03-07T15:48:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.