# ERR Kafka (topic=filebeat-test-logmiss30): dropping too large message of size 3463 **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 25, 2018, 8:48am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003 "2018-01-25T08:48:14Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\ **Post date:** [January 25, 2018, 8:48am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/1 "2018-01-25T08:48:15Z")
I use filebeat to publish log to kafka, but get "2018/01/25 08:39:54.536086 client.go:203: ERR Kafka (topic=filebeat-test-logmiss30): dropping too large message of size 3463." configurations: 1. Filebeat(6.1.2) output.kafka: max\_message\_bytes: 1000000 version: 0.11.0.0 2. Kafka(0.11.0.1) kakfka broker config message.max.bytes: 1000012 Note: Total of 27000 logs, each log size is the same(3k), but half of them are published successful, half of them are dropped. ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [January 25, 2018, 9:55am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/2 "2018-01-25T09:55:36Z")
Can you try setting the `output.kafka.bulk_max_size` setting in filebeat to 100 to see if the problem is gone? It seems that Kafka is applying the message.max.bytes limit to the whole batch and not to individual messages. ---
**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\ **Post date:** [January 26, 2018, 3:29am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/3 "2018-01-26T03:29:15Z")
It works, output.kafka bulk\_max\_size times max\_message\_bytes should smaller than Kafka message.max.bytes. ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [January 26, 2018, 2:42pm UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/4 "2018-01-26T14:42:44Z")
We've been reviewing the code and it seems my original diagnosis is wrong. Changing the `bulk_max_size` may just be mitigating the problem out of sheer luck, but also reducing the throughput. Is not a good fix. We would like to investigate this problem further, can you provide us with: - Full `filebeat.yml` - Full Kafka broker configuration - tcpdump of Kafka traffic when this problem happens ---
**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\ **Post date:** [January 29, 2018, 5:07am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/5 "2018-01-29T05:07:37Z")
**Problem Situation** **FileBeat** 1.filebeat-kafka.yml ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/e/feda7609325dd592123b84cde7ff6db06dd384e5.png) 2.config/\*.yml ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c44ad086a5029541540a66a802db8130e9975713.png) **Kafka** kafka broker configurations broker.id=97 listeners=... num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=... num.partitions=3 default.replication.factor=2 num.recovery.threads.per.data.dir=1 log.retention.hours=24 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=... zookeeper.connection.timeout.ms=6000 reserved.broker.max.id=2147483647 **Error log** 2018/01/29 03:53:29.703780 client.go:203: ERR Kafka (topic=filebeat-test-0129): dropping too large message of size 3446. 2018/01/29 03:53:29.703802 client.go:203: ERR Kafka (topic=filebeat-test-0129): dropping too large message of size 3446. 2018/01/29 03:53:29.703811 client.go:203: ERR Kafka (topic=filebeat-test-0129): dropping too large message of size 3446. **tcpdump** sudo tcpdump -i em1 dst port 20099 -A ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/4/14837936891c25e789fe9d4387d643f771571f04.png) ===================================================================================== **Fix** To ensure that "output.kafka bulk\_max\_size times max\_message\_bytes should smaller than Kafka message.max.bytes", I set the topic-level max.message.bytes to 52428800. **FileBeat** ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/b/dbc297921afb7e3aab05a17f44d443c36e499502.png) **Kafka** ./kafka-configs.sh --zookeeper ... --entity-type topics --entity-name filebeat-test-0129 --describe Configs for topic 'filebeat-test-0129' are max.message.bytes=52428800 Throughput: 25K~30K record/s, 80 Mb/s (log size is 3k) ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [January 29, 2018, 1:59pm UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/6 "2018-01-29T13:59:26Z")
Please, can you post a packet capture file (pcap) instead of a screenshot? You can use tcpdump's `-w filename.pcap` option or use Wireshark's `File -> Save`. ---
**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\ **Post date:** [January 30, 2018, 5:02am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/7 "2018-01-30T05:02:04Z")
[https://drive.google.com/file/d/1x\_IUe0Mj4O59IQesijPGoMvcoyDlvsN4/view?usp=sharing](https://drive.google.com/file/d/1x_IUe0Mj4O59IQesijPGoMvcoyDlvsN4/view?usp=sharing) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 27, 2018, 5:02am UTC](https://discuss.elastic.co/t/err-kafka-topic-filebeat-test-logmiss30-dropping-too-large-message-of-size-3463/117003/8 "2018-02-27T05:02:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.