filebeat: 7.4.2
ES:7.4.2
filebeat.yml:
filebeat.inputs:
- type: kafka
hosts:
- wn0-xxx.internal.chinacloudapp.cn:9092
- wn1-xxx.internal.chinacloudapp.cn:9092
- wn2-xxx.internal.chinacloudapp.cn:9092
topics: ["k8s-pod-logs-qa-kafka-topic"]
group_id: "fb-k8s-pod-qa-group"
output.elasticsearch:
hosts: ["127.0.0.1:9200"]
index: "kafka-application-kube-%{+yyyy.MM.dd}"
pipeline: kafka-kubernetes-daemon
xpack.monitoring:
enabled: true
setup.ilm.enabled: false
setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
2019-12-10T08:33:54.799Z INFO kafka/log.go:53 consumer/broker/1001 abandoned subscription to k8s-pod-logs-qa-kafka-topic/1 because consuming was taking too long
2019-12-10T08:33:55.343Z INFO kafka/log.go:53 consumer/broker/1002 added subscription to k8s-pod-logs-qa-kafka-topic/2
2019-12-10T08:33:55.371Z INFO kafka/log.go:53 consumer/broker/1002 abandoned subscription to k8s-pod-logs-qa-kafka-topic/2 because consuming was taking too long
2019-12-10T08:33:56.230Z INFO kafka/log.go:53 consumer/broker/1003 added subscription to k8s-pod-logs-qa-kafka-topic/0
2019-12-10T08:33:56.307Z INFO kafka/log.go:53 consumer/broker/1003 abandoned subscription to k8s-pod-logs-qa-kafka-topic/0 because consuming was taking too long
I think, when output to ES, it's take a long time, caused by "Consumer.MaxProcessingTime" in here is too short, but I can't setting this parameter in kafka plugin,how to resove it?
When output to console, it's ok!