Input plugin: kafka, error: "abandoned subscription to k8s-pod-logs-kafka-topic/0 because consuming was taking too long"

Elastic Stack Beats
1

filebeat: 7.4.2
ES:7.4.2
filebeat.yml:

filebeat.inputs:
- type: kafka
  hosts:
    - wn0-xxx.internal.chinacloudapp.cn:9092
    - wn1-xxx.internal.chinacloudapp.cn:9092
    - wn2-xxx.internal.chinacloudapp.cn:9092
  topics: ["k8s-pod-logs-qa-kafka-topic"]
  group_id: "fb-k8s-pod-qa-group"

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  index: "kafka-application-kube-%{+yyyy.MM.dd}"
  pipeline: kafka-kubernetes-daemon

xpack.monitoring:
  enabled: true

setup.ilm.enabled: false
setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"

2019-12-10T08:33:54.799Z        INFO    kafka/log.go:53 consumer/broker/1001 abandoned subscription to k8s-pod-logs-qa-kafka-topic/1 because consuming was taking too long

2019-12-10T08:33:55.343Z        INFO    kafka/log.go:53 consumer/broker/1002 added subscription to k8s-pod-logs-qa-kafka-topic/2

2019-12-10T08:33:55.371Z        INFO    kafka/log.go:53 consumer/broker/1002 abandoned subscription to k8s-pod-logs-qa-kafka-topic/2 because consuming was taking too long

2019-12-10T08:33:56.230Z        INFO    kafka/log.go:53 consumer/broker/1003 added subscription to k8s-pod-logs-qa-kafka-topic/0

2019-12-10T08:33:56.307Z        INFO    kafka/log.go:53 consumer/broker/1003 abandoned subscription to k8s-pod-logs-qa-kafka-topic/0 because consuming was taking too long

I think, when output to ES, it's take a long time, caused by "Consumer.MaxProcessingTime" in here is too short, but I can't setting this parameter in kafka plugin,how to resove it?

When output to console, it's ok!

2

Hi!

MaxProcessingTime might be the problem here as you indicate, thanks! I will prepare a patch adding this setting in our side.

However, do you observe anything else in Filebeat's logs after these specific logs? Does Filebeat continue running after this?

Thanks!

3

It keeps running, but the log keeps repeating. it just tries to resubscribe the topic.

4

Hey see,

5

Hi, ChrsMark, good job! Thanks

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.