# Kafka output dropping messages due to size, but doesn't increment the "failed" counter **URL:** \ **Category:** Beats\ **Created:** [December 1, 2019, 7:44pm UTC](https://discuss.elastic.co/t/kafka-output-dropping-messages-due-to-size-but-doesnt-increment-the-failed-counter/210081 "2019-12-01T19:44:29Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![imriz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imriz/32/58591_2.png) [@imriz](https://discuss.elastic.co/u/imriz)\ **Post date:** [December 1, 2019, 7:44pm UTC](https://discuss.elastic.co/t/kafka-output-dropping-messages-due-to-size-but-doesnt-increment-the-failed-counter/210081/1 "2019-12-01T19:44:29Z")
Not sure yet if this is a bug, or intentional behavior - I've noticed in my logs that sometimes Filebeat's Kafka output will drop messages due to size error from the brokerm, but won't increment the "failed" counter ([https://github.com/elastic/beats/blob/master/libbeat/outputs/kafka/client.go#L247](https://github.com/elastic/beats/blob/master/libbeat/outputs/kafka/client.go#L247)). It seems this was introduced here: [https://github.com/elastic/beats/pull/2737](https://github.com/elastic/beats/pull/2737) as a result of the discussion in [Filebeat 5.0beta1 with kafka output: multiplication of log lines](https://discuss.elastic.co/t/filebeat-5-0beta1-with-kafka-output-multiplication-of-log-lines/62513/12) The end result is that apparently one can lose messages without any alert ever firing (since the failures stats are not being incremented). This seems like the wrong behavior, and I think this should be accounted under "failed" or "dropped", so one could be alerted. ---
**Author:** ![imriz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imriz/32/58591_2.png) [@imriz](https://discuss.elastic.co/u/imriz)\ **Post date:** [December 10, 2019, 3:10pm UTC](https://discuss.elastic.co/t/kafka-output-dropping-messages-due-to-size-but-doesnt-increment-the-failed-counter/210081/2 "2019-12-10T15:10:15Z")
For the sake of future googlers, this was indeed a bug, which was fixed: > > > Not sure yet if this is a bug, or intentional behavior - > I've noticed in my logs that sometimes Filebeat's Kafka output... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 7, 2020, 5:11pm UTC](https://discuss.elastic.co/t/kafka-output-dropping-messages-due-to-size-but-doesnt-increment-the-failed-counter/210081/3 "2020-01-07T17:11:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.