# ERR Kafka (topic={topic}): dropping too large message of size

**URL:** <https://discuss.elastic.co/t/err-kafka-topic-topic-dropping-too-large-message-of-size/109560>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 29, 2017, 10:05am UTC](https://discuss.elastic.co/t/err-kafka-topic-topic-dropping-too-large-message-of-size/109560 "2017-11-29T10:05:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![olololll](https://avatars.discourse-cdn.com/v4/letter/o/ac91a4/32.png) [@olololll](https://discuss.elastic.co/u/olololll)\
**Post date:** [November 29, 2017, 10:05am UTC](https://discuss.elastic.co/t/err-kafka-topic-topic-dropping-too-large-message-of-size/109560/1 "2017-11-29T10:05:59Z")

</div>

Im using Filebeat(5.6.3) connecting to Kafka(0.10.2.1).

However, i got an error as below in filebeat:

`> 2017-11-29T17:12:28+08:00 ERR Kafka (topic=message): dropping too large message of large size 3038538.`

It means i have some messages discarded while going into Kakfa because of large message size.

Then I try to fix this by my Filebeat yml file, i added a line under `filebeat.prospectors:`

`max_bytes: 10485760`

[https://www.elastic.co/guide/en/beats/filebeat/5.6/configuration-filebeat-options.html#max-message-size](https://www.elastic.co/guide/en/beats/filebeat/5.6/configuration-filebeat-options.html#max-message-size)

But not work, the error still occur.

Where is the config that i should correct for this error? Or should i amend the config file in Kafka?

Thank you.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 29, 2017, 2:32pm UTC](https://discuss.elastic.co/t/err-kafka-topic-topic-dropping-too-large-message-of-size/109560/2 "2017-11-29T14:32:36Z")

</div>

1. Kafka itself enforces a limit on message sizes. You will have to update the kafka brokers to allow for bigger messages.

2. beats kafka output checks the JSON encoded event size. If the size hits the limit in the output, the event is dropped

3. the `max_bytes` setting sets the log message size. The encoded event can be much bigger, due to additional fields + string escaping. That is `max_bytes` should be somewhat smaller then the max event size allowed by kafka and the kafka output in beats.

If you are fine with the default event limit in kafka, try reducing `max_bytes` somewhat more.

---

<div class="post-metadata">

**Author:** ![olololll](https://avatars.discourse-cdn.com/v4/letter/o/ac91a4/32.png) [@olololll](https://discuss.elastic.co/u/olololll)\
**Post date:** [November 30, 2017, 9:03am UTC](https://discuss.elastic.co/t/err-kafka-topic-topic-dropping-too-large-message-of-size/109560/3 "2017-11-30T09:03:25Z")

</div>

Thanks!  
My problem was solved by 2 configurations:

1. filebeat yml file, under "output.kafka:",

> max\_message\_bytes:

1. kafka server properties,

> message.max.bytes=

And my logstash conf behind was also amended:

> input{  
> kafka{  
> max\_partition\_fetch\_bytes =\>" "  
> }  
> }

hope this could help someone.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2017, 9:03am UTC](https://discuss.elastic.co/t/err-kafka-topic-topic-dropping-too-large-message-of-size/109560/4 "2017-12-28T09:03:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.