# Filebeat 5.0beta1 with kafka output: multiplication of log lines

**URL:** <https://discuss.elastic.co/t/filebeat-5-0beta1-with-kafka-output-multiplication-of-log-lines/62513>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 7, 2016, 5:17pm UTC](https://discuss.elastic.co/t/filebeat-5-0beta1-with-kafka-output-multiplication-of-log-lines/62513 "2016-10-07T17:17:17Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 10, 2016, 11:31am UTC](https://discuss.elastic.co/t/filebeat-5-0beta1-with-kafka-output-multiplication-of-log-lines/62513/12 "2016-10-10T11:31:18Z")

</div>

> [@ruflin](#):
>
> About the max size: What is the message returned by Kafka if that happens. In case it is an error / no acked this could explain the repeated events as filebeat constantly resends the event and kafka potential stores a "subset" of the event but not the full event.

Digging into the code, this can indeed be a problem. Doc says message is dropped, but code returns an error `sarama.ErrMessageSizeTooLarge`. While sarama would only drop this message, libbeat aggregates all errors for a batch, potentially forcing kafka to resend the complete batch.

@psychonaut maybe you can create an [github issue](https://github.com/elastic/beats/issues) about batches being resend if some event in the batch is too big. I'd really like to mark the issue for the [Pioneer Program](https://www.elastic.co/blog/elastic-pioneer-program).

I will prepare a fix on master for just dropping too large messages with an error messsage (output can not tell which fields need to be shortened). Fix should be to adapt sizes in kafka itself + in filebeat.yml kafka output section and/or multiline. This is unfortunately fully up to the user. As these big events are normally traces, no one wants to drop them, but kafka settings can force producers to do so.  
I'm thinking about adding a dead-letter queue for a while (e.g. for this use-case), but definitely not in 5.0 release.

---

_[View the full topic](https://discuss.elastic.co/t/filebeat-5-0beta1-with-kafka-output-multiplication-of-log-lines/62513)._