# Single unavailable Kafka topic blocks delivery to other available topics

**URL:** <https://discuss.elastic.co/t/single-unavailable-kafka-topic-blocks-delivery-to-other-available-topics/152189>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 12, 2018, 8:37am UTC](https://discuss.elastic.co/t/single-unavailable-kafka-topic-blocks-delivery-to-other-available-topics/152189 "2018-10-12T08:37:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tsuhachev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsuhachev/32/36440_2.png) [@tsuhachev](https://discuss.elastic.co/u/tsuhachev)\
**Post date:** [October 12, 2018, 8:37am UTC](https://discuss.elastic.co/t/single-unavailable-kafka-topic-blocks-delivery-to-other-available-topics/152189/1 "2018-10-12T08:37:25Z")

</div>

filebeat version 5.4.1 (amd64), libbeat 5.4.1 on CentOs 7  
filebeat.prospectors:  
- input\_type: log  
backoff: 200ms  
max\_backoff: 500ms  
scan\_frequency: 2s  
paths:  
- /logs/login-events.log  
document\_type: name  
fields:  
instance: auth-service-1  
log\_type: auth\_login\_events  
topic: topic\_name  
include\_lines: ['.\*']  
json.message\_key: uniqueKey  
json.keys\_under\_root: true  
json.overwrite\_keys: true

I have filebeat delivering messages from 2 files to 2 different Kafka topics accordingly  
If for any reason one of Kafka topic is unavailable, filebeat stops delivery to the existing topic as well.  
Instead it proactively tries to re-connect to the unavailable topic.

Interesting that if I restart filebeat it will send to the existing topic all the messages available since the last restart, but will ignore all new messages

Do I miss some configuration? Though to me it looks more like a bug.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 16, 2018, 8:48pm UTC](https://discuss.elastic.co/t/single-unavailable-kafka-topic-blocks-delivery-to-other-available-topics/152189/2 "2018-10-16T20:48:52Z")

</div>

Filebeat maintains an internal event queue. The size of the queue is bounded. Once the queue is full, no more events can be read. Events must be ACKed in order, so to guarantee correct state updates. One failing topic can block the queue, as the queue will run full.

As the queue is a shared resource, you have some indirect coupling between these topics in filebeat. For full independent processing you will need 2 filebeat processes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 13, 2018, 8:53pm UTC](https://discuss.elastic.co/t/single-unavailable-kafka-topic-blocks-delivery-to-other-available-topics/152189/3 "2018-11-13T20:53:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.