# Filebeat splits message after 16k **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 13, 2018, 10:43am UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718 "2018-03-13T10:43:06Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![paltryeffort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltryeffort/32/28717_2.png) [@paltryeffort](https://discuss.elastic.co/u/paltryeffort)\ **Post date:** [March 13, 2018, 10:43am UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/1 "2018-03-13T10:43:06Z")
``` # filebeat 6.2.2 filebeat.prospectors: - paths: ["/var/lib/docker/containers/*/*-json.log"] harvester_buffer_size: 65536 json.message_key: log json.keys_under_root: true json.add_error_key: true fields_under_root: true processors: - add_docker_metadata: ~ harvester_buffer_size: 65536 output.file: path: "/tmp/filebeat" filename: filebeat ``` - The docker container outputs a single line in valid json (about 18k) - The value of the "log" message key is also a single line in valid json - The message seems to be cut off at about 16k or a bit above (depends if you count the backslashes for escaping) - A second message gets created with the remaining part of the message including full decoration (docker meta data, additional fields etc) - Looks like filebeat splits the message into 2 separate ones - harvester\_buffer\_size has no effect - removing the json.\* options has no effect ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [March 13, 2018, 11:12am UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/2 "2018-03-13T11:12:12Z")
Hi @paltryeffort, We introduced the [docker prospector](https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html#config-containers), it handles JSON decoding and timestamp retrieval for you. Could you confirm if this is still happening with it? Conf would look like this: ```auto filebeat.prospectors: - type: docker containers.ids: - "*" processors: - add_docker_metadata: ~ ``` ---
**Author:** ![paltryeffort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltryeffort/32/28717_2.png) [@paltryeffort](https://discuss.elastic.co/u/paltryeffort)\ **Post date:** [March 13, 2018, 11:50am UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/3 "2018-03-13T11:50:51Z")
Hi @exekias , thanks for the fast response. Using your config the message gets cut off at the exact same position. I tried it with and without the harvester\_buffer\_size option. ---
**Author:** ![jsamphillips](https://avatars.discourse-cdn.com/v4/letter/j/67e7ee/32.png) [@jsamphillips](https://discuss.elastic.co/u/jsamphillips)\ **Post date:** [March 15, 2018, 2:51pm UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/4 "2018-03-15T14:51:13Z")
We're having this exact same issue. Nothing I have tweaked seems to change this behavior. Interested to see what the solution is ---
**Author:** ![paltryeffort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltryeffort/32/28717_2.png) [@paltryeffort](https://discuss.elastic.co/u/paltryeffort)\ **Post date:** [March 15, 2018, 4:32pm UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/5 "2018-03-15T16:32:12Z")
How to reproduce: use my config, start a container and run the following: ``` #!/bin/bash key="somerandomkey_" value="somerandomvalue_" echo -n '{' for i in $(seq 420); do echo -n "\"${key}${i}\":\"${value}${i}\"," done echo '"lastkey":"end"}' ``` This produces a valid json output. For me it cuts off at "somerand" and a new message continues with "omkey\_396". ---
**Author:** ![jsamphillips](https://avatars.discourse-cdn.com/v4/letter/j/67e7ee/32.png) [@jsamphillips](https://discuss.elastic.co/u/jsamphillips)\ **Post date:** [March 20, 2018, 6:25pm UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/6 "2018-03-20T18:25:37Z")
Bump on this. @exekias is there anything else we can try to do to resolve this issue? ---
**Author:** ![bitsofinfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bitsofinfo/32/459_2.png) [@bitsofinfo](https://discuss.elastic.co/u/bitsofinfo)\ **Post date:** [March 20, 2018, 7:24pm UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/7 "2018-03-20T19:24:48Z")
issued added here, please comment on it with your experiences to get some traction going, its definitely a deal breaker! > ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 22, 2018, 1:48pm UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/8 "2018-03-22T13:48:24Z")
Thanks for opening the issue. We will try to find some time to look into it. ---
**Author:** ![bitsofinfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bitsofinfo/32/459_2.png) [@bitsofinfo](https://discuss.elastic.co/u/bitsofinfo)\ **Post date:** [March 22, 2018, 2:12pm UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/9 "2018-03-22T14:12:40Z")
Pretty urgent for us, may have to use else due to this, for example apps that generate stack traces or if you are consuming modsecurity audit logs, sizes of entries like this are pretty typical and w/ fbeats we are not sure how to re-correlate this data in logstash? any suggestions from elastic for us and others dealing with this? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 26, 2018, 9:01am UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/10 "2018-03-26T09:01:20Z")
For everyone coming to this thread, discussion continues in [https://github.com/elastic/beats/issues/6605](https://github.com/elastic/beats/issues/6605) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 23, 2018, 9:01am UTC](https://discuss.elastic.co/t/filebeat-splits-message-after-16k/123718/11 "2018-04-23T09:01:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.