Input plugin: kafka, error: "abandoned subscription to k8s-pod-logs-kafka-topic/0 because consuming was taking too long"

filebeat: 7.4.2

- type: kafka
  topics: ["k8s-pod-logs-qa-kafka-topic"]
  group_id: "fb-k8s-pod-qa-group"

  hosts: [""]
  index: "kafka-application-kube-%{+yyyy.MM.dd}"
  pipeline: kafka-kubernetes-daemon

  enabled: true

setup.ilm.enabled: false "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
2019-12-10T08:33:54.799Z        INFO    kafka/log.go:53 consumer/broker/1001 abandoned subscription to k8s-pod-logs-qa-kafka-topic/1 because consuming was taking too long

2019-12-10T08:33:55.343Z        INFO    kafka/log.go:53 consumer/broker/1002 added subscription to k8s-pod-logs-qa-kafka-topic/2

2019-12-10T08:33:55.371Z        INFO    kafka/log.go:53 consumer/broker/1002 abandoned subscription to k8s-pod-logs-qa-kafka-topic/2 because consuming was taking too long

2019-12-10T08:33:56.230Z        INFO    kafka/log.go:53 consumer/broker/1003 added subscription to k8s-pod-logs-qa-kafka-topic/0

2019-12-10T08:33:56.307Z        INFO    kafka/log.go:53 consumer/broker/1003 abandoned subscription to k8s-pod-logs-qa-kafka-topic/0 because consuming was taking too long

I think, when output to ES, it's take a long time, caused by "Consumer.MaxProcessingTime" in here is too short, but I can't setting this parameter in kafka plugin,how to resove it?

When output to console, it's ok!


MaxProcessingTime might be the problem here as you indicate, thanks! I will prepare a patch adding this setting in our side.

However, do you observe anything else in Filebeat's logs after these specific logs? Does Filebeat continue running after this?


It keeps running, but the log keeps repeating. it just tries to resubscribe the topic.

Hey see,

Hi, ChrsMark, good job! Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.