# How to recover from split log messages

**URL:** https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579
**Category:** Kibana
**Created:** [September 2, 2022, 6:04pm UTC](https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579 "2022-09-02T18:04:56Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)
#### Post date: [September 2, 2022, 6:04pm UTC](https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579/1 "2022-09-02T18:04:56Z")

</div>

There appears to be a years-long-standing issue with large ( longer than 16KB) messages getting split into parts and appearing on Kibana in multiple lines. Such long messages typically include Java exception stack traces.

As much as I have searched I have not found a filter that can concatenate those parts and make them appear as a whole on a single log entry, where the message can be properly parsed and indexed. I have tried a few filters of my own, with little success.

Please help if you are aware of a solution.

Thanks

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [September 2, 2022, 6:26pm UTC](https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579/2 "2022-09-02T18:26:58Z")

</div>

Can you give more context of your issue?

Kibana only shows the messages that are in Elasticsearch, if for example you have a Java Exception Stack Trace split in multiple documents, the issue happens during the ingestion of the data and can be solved using a multiline codec that exists both in Logstash and Filebeat.

---

<div class="post-metadata">

### Author: ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)
#### Post date: [September 2, 2022, 7:29pm UTC](https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579/3 "2022-09-02T19:29:06Z")

</div>

Thanks for responding!  
I'm using EFK, i.e. FluentD, it' s running as a Helm chart on an AWS EKS cluster. This link describes the problem in some detail, and contains a proposed solution by user=dsx0, one that I tried but didn't work for me. EFK is only a very small part of my job, so I'm not well-versed in it. Still it's up to me to fix this nagging problem.

> <https://github.com/moby/moby/issues/34620>
>
> \<!--
> If you are reporting a new issue, make sure that we do not have any duplic…ates
> already open. You can ensure this by searching the issue list for this
> repository. If there is a duplicate, please close your issue and add a comment
> to the existing issue instead.
> 
> If you suspect your issue is a bug, please edit your issue description to
> include the BUG REPORT INFORMATION shown below. If you fail to provide this
> information within 7 days, we cannot debug your issue and will close it. We
> will, however, reopen it if you later provide the information.
> 
> For more information about reporting issues, see
> https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues
> 
> \---------------------------------------------------
> GENERAL SUPPORT INFORMATION
> \---------------------------------------------------
> 
> The GitHub issue tracker is for bug reports and feature requests.
> General support can be found at the following locations:
> 
> \- Docker Support Forums - https://forums.docker.com
> \- IRC - irc.freenode.net #docker channel
> \- Post a question on StackOverflow, using the Docker tag
> 
> \---------------------------------------------------
> BUG REPORT INFORMATION
> \---------------------------------------------------
> Use the commands below to provide key information from your environment:
> You do NOT have to include this information if this is a FEATURE REQUEST
> \--\>
> 
> \*\*Description\*\*
> Log message is split into chunks about ~16374 characters when using fluentd driver with docker.
> I tested the same log message using fluent-logger-python and it works perfectly good. \[Details\](https://github.com/fluent/fluentd/issues/1669)
> 
> \*\*Configuration, log message example and output log\*\*:
> \[fluent.conf\](https://gist.github.com/ymatsiuk/3bb0ce001e771fd02b70a56e6bc053ff#file-fluent-conf)
> \[new.json\](https://gist.github.com/ymatsiuk/3bb0ce001e771fd02b70a56e6bc053ff#file-new-json)
> \[output.log\](https://gist.github.com/ymatsiuk/3bb0ce001e771fd02b70a56e6bc053ff#file-output-log)
> \[Dockerfile\](https://gist.github.com/ymatsiuk/cf02d880a7b0b2ef69c8a7ffd6ea71b1#file-dockerfile)
> 
> \*\*Steps to reproduce the issue:\*\*
> 1. \`docker run -it -p 24224:24224 -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf ymatsiuk/fluentd-test:latest\`
> 2. \`docker run -it --log-driver=fluentd --log-opt tag=docker.{{.ID}} -v /Users/ymatsiuk/test:/root/test alpine cat /root/test/new.json\`
> 
> \*\*Describe the results you received:\*\*
> Log message is split into chunks which makes it more complicated to investigate through this log. In my case it was a huge .json which contains sensitive data and useful for debugging, but I have to replace it with just simple long text. Original .json was split into 9 chunks and created a mess in kibana.
> 
> \*\*Describe the results you expected:\*\*
> Message should remain solid 
> 
> \*\*Output of \`docker version\`:\*\*
> 
> \`\`\`
> Client:
> Version: 17.07.0-ce-rc3
> API version: 1.31
> Go version: go1.8.3
> Git commit: 665d244
> Built: Thu Aug 17 00:56:49 2017
> OS/Arch: darwin/amd64
> 
> Server:
> Version: 17.07.0-ce-rc3
> API version: 1.31 (minimum version 1.12)
> Go version: go1.8.3
> Git commit: 665d244
> Built: Thu Aug 17 01:32:35 2017
> OS/Arch: linux/amd64
> Experimental: true
> \`\`\`
> 
> \*\*Output of \`docker info\`:\*\*
> 
> \`\`\`
> Containers: 0
> Running: 0
> Paused: 0
> Stopped: 0
> Images: 0
> Server Version: 17.07.0-ce-rc3
> Storage Driver: overlay2
> Backing Filesystem: extfs
> Supports d\_type: true
> Native Overlay Diff: true
> Logging Driver: json-file
> Cgroup Driver: cgroupfs
> Plugins:
> Volume: local
> Network: bridge host ipvlan macvlan null overlay
> Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
> Swarm: inactive
> Runtimes: runc
> Default Runtime: runc
> Init Binary: docker-init
> containerd version: 3addd840653146c90a254301d6c3a663c7fd6429
> runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
> init version: 949e6fa
> Security Options:
> seccomp
> Profile: default
> Kernel Version: 4.9.41-moby
> Operating System: Alpine Linux v3.5
> OSType: linux
> Architecture: x86\_64
> CPUs: 4
> Total Memory: 1.952GiB
> Name: moby
> ID: XZJL:MK7G:BLBH:GX7K:LG3A:FYIE:LGK3:DGQX:KCLO:YD7A:LH4O:EVMU
> Docker Root Dir: /var/lib/docker
> Debug Mode (client): false
> Debug Mode (server): true
> File Descriptors: 19
> Goroutines: 31
> System Time: 2017-08-24T07:18:02.582487431Z
> EventsListeners: 1
> No Proxy: \*.local, 169.254/16
> Registry: https://index.docker.io/v1/
> Experimental: true
> Insecure Registries:
> 127.0.0.0/8
> Live Restore Enabled: false
> \`\`\`

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [September 2, 2022, 10:20pm UTC](https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579/4 "2022-09-02T22:20:47Z")

</div>

Unfortunately the issue is not related to Kibana or any tool in the Elastic stack.

Your messages are being splitted before they arrive in Elasticsearch and each part of the message is treated as an individual document.

The way to solve this is to combine the parts of the splitted message and reconstruct it **before** sending to Elasticsearch, both Logstash and Filebeat, which are part of Elastic Stack have a multiline codec that helps you reconstruct the messages, but I have no idea if FluentD has something similar.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 30, 2022, 10:21pm UTC](https://discuss.elastic.co/t/how-to-recover-from-split-log-messages/313579/5 "2022-09-30T22:21:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.