Kafka output dropping messages due to size, but doesn't increment the "failed" counter

Elastic Stack Beats
1

Not sure yet if this is a bug, or intentional behavior -
I've noticed in my logs that sometimes Filebeat's Kafka output will drop messages due to size error from the brokerm, but won't increment the "failed" counter (https://github.com/elastic/beats/blob/master/libbeat/outputs/kafka/client.go#L247).
It seems this was introduced here: https://github.com/elastic/beats/pull/2737 as a result of the discussion in Filebeat 5.0beta1 with kafka output: multiplication of log lines

The end result is that apparently one can lose messages without any alert ever firing (since the failures stats are not being incremented). This seems like the wrong behavior, and I think this should be accounted under "failed" or "dropped", so one could be alerted.

2

For the sake of future googlers, this was indeed a bug, which was fixed:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.