# Input plugin: kafka, error: "abandoned subscription to k8s-pod-logs-kafka-topic/0 because consuming was taking too long"

**URL:** <https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 10, 2019, 9:35am UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259 "2019-12-10T09:35:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![111258](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111258/32/59099_2.png) [@111258](https://discuss.elastic.co/u/111258)\
**Post date:** [December 10, 2019, 9:35am UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259/1 "2019-12-10T09:35:21Z")

</div>

filebeat: 7.4.2  
ES:7.4.2  
filebeat.yml:

```auto
filebeat.inputs:
- type: kafka
  hosts:
    - wn0-xxx.internal.chinacloudapp.cn:9092
    - wn1-xxx.internal.chinacloudapp.cn:9092
    - wn2-xxx.internal.chinacloudapp.cn:9092
  topics: ["k8s-pod-logs-qa-kafka-topic"]
  group_id: "fb-k8s-pod-qa-group"

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  index: "kafka-application-kube-%{+yyyy.MM.dd}"
  pipeline: kafka-kubernetes-daemon

xpack.monitoring:
  enabled: true

setup.ilm.enabled: false
setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"

```

```auto
2019-12-10T08:33:54.799Z INFO kafka/log.go:53 consumer/broker/1001 abandoned subscription to k8s-pod-logs-qa-kafka-topic/1 because consuming was taking too long

2019-12-10T08:33:55.343Z INFO kafka/log.go:53 consumer/broker/1002 added subscription to k8s-pod-logs-qa-kafka-topic/2

2019-12-10T08:33:55.371Z INFO kafka/log.go:53 consumer/broker/1002 abandoned subscription to k8s-pod-logs-qa-kafka-topic/2 because consuming was taking too long

2019-12-10T08:33:56.230Z INFO kafka/log.go:53 consumer/broker/1003 added subscription to k8s-pod-logs-qa-kafka-topic/0

2019-12-10T08:33:56.307Z INFO kafka/log.go:53 consumer/broker/1003 abandoned subscription to k8s-pod-logs-qa-kafka-topic/0 because consuming was taking too long

```

I think, when output to ES, it's take a long time, caused by "Consumer.MaxProcessingTime" in [here](https://github.com/Shopify/sarama/blob/c82acaf7831341c57794be79578431c927540840/consumer.go#L442) is too short, but I can't setting this parameter in kafka plugin,how to resove it?

When output to console, it's ok!

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [December 10, 2019, 12:47pm UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259/2 "2019-12-10T12:47:34Z")

</div>

Hi!

`MaxProcessingTime` might be the problem here as you indicate, thanks! I will prepare a patch adding this setting in our side.

However, do you observe anything else in Filebeat's logs after these specific logs? Does Filebeat continue running after this?

Thanks!

---

<div class="post-metadata">

**Author:** ![111258](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111258/32/59099_2.png) [@111258](https://discuss.elastic.co/u/111258)\
**Post date:** [December 10, 2019, 3:51pm UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259/3 "2019-12-10T15:51:40Z")

</div>

It keeps running, but the log keeps repeating. it just tries to resubscribe the topic.

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [December 11, 2019, 8:31am UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259/4 "2019-12-11T08:31:15Z")

</div>

Hey see,

> <https://github.com/elastic/beats/pull/15031>

---

<div class="post-metadata">

**Author:** ![111258](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111258/32/59099_2.png) [@111258](https://discuss.elastic.co/u/111258)\
**Post date:** [December 12, 2019, 1:54am UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259/5 "2019-12-12T01:54:50Z")

</div>

Hi, ChrsMark, good job! Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2020, 2:03am UTC](https://discuss.elastic.co/t/input-plugin-kafka-error-abandoned-subscription-to-k8s-pod-logs-kafka-topic-0-because-consuming-was-taking-too-long/211259/6 "2020-01-09T02:03:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.