# Filebeat cannot merge partial events

**URL:** <https://discuss.elastic.co/t/filebeat-cannot-merge-partial-events/274119>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [May 26, 2021, 9:14pm UTC](https://discuss.elastic.co/t/filebeat-cannot-merge-partial-events/274119 "2021-05-26T21:14:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![delta9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/delta9/32/89309_2.png) [@delta9](https://discuss.elastic.co/u/delta9)\
**Post date:** [May 26, 2021, 9:14pm UTC](https://discuss.elastic.co/t/filebeat-cannot-merge-partial-events/274119/1 "2021-05-26T21:14:48Z")

</div>

Hi all.

We use filebeat to collect events from pods in k8s cluster. To do this we run it as `daemonset` and mounts `/var/lib/docker/containers`, `/var/log/pods` and `/var/log/containers` from cluster nodes into filebeat pods.

The configuration file for filebeat:

```auto
- type: container
  paths:
    - /var/log/containers/*.log
  close_inactive: 10m
  close_removed: true
  ignore_older: 12m
  clean_inactive: 15m
  clean_removed: true
  multiline.pattern: '^(\{s{0,1}\")'.
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 10s
  harvester_buffer_size: 65536
  scan_frequency: 1s
  partial: true
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
    - decode_json_fields:
        fields: ['message']
        target: ''
        max_depth: 3
        add_error_key: true

```

Docker settings:

```auto
    {
            "log-driver": { "json-file",
            { "log-opts": {
                    { "max-file": "5",
                    { "max-size": "10m"
            },
            { "max-concurrent-downloads": 3
    }

```

We have the following issues:  
Applications pods produce events that can be either a single long line (up to 50Kb) or multiline (up to 100Kb total). Long lines are cut by docker at 16Kb and 10% of the time filebeat does not merge these lines into a single event.  
Also multiline events will not merge if parts of the entries are in different logs (podname.log and podname.log.1 due to rotation)

Is it a bug or misconfigured?

**filebeat - [docker.elastic.co/beats/filebeat-oss:7.9.3](http://docker.elastic.co/beats/filebeat-oss:7.9.3)**  
**docker - v19.03.13**  
**kubernetes - v1.19.8**

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [May 27, 2021, 7:55am UTC](https://discuss.elastic.co/t/filebeat-cannot-merge-partial-events/274119/2 "2021-05-27T07:55:39Z")

</div>

> Long lines are cut by docker at 16Kb

Yes, there was a thread about it: [Join Docker log lines when they are split because of size by exekias · Pull Request #6967 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/6967) . I think you can try to experiment with the docker input. It's a different method.

> Is it a bug or misconfigured?

I'm not sure actually without any problematic samples. You need to review entries which are not merged (maybe there regexp is not bulletproof).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 24, 2021, 9:55am UTC](https://discuss.elastic.co/t/filebeat-cannot-merge-partial-events/274119/3 "2021-06-24T09:55:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.