# Filebeat start endless reconnectiosn when connect error happen to kafka cluster

**URL:** <https://discuss.elastic.co/t/filebeat-start-endless-reconnectiosn-when-connect-error-happen-to-kafka-cluster/56241>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 24, 2016, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-start-endless-reconnectiosn-when-connect-error-happen-to-kafka-cluster/56241 "2016-07-24T14:41:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![DeeeFOX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deeefox/32/49566_2.png) [@DeeeFOX](https://discuss.elastic.co/u/DeeeFOX)\
**Post date:** [July 24, 2016, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-start-endless-reconnectiosn-when-connect-error-happen-to-kafka-cluster/56241/1 "2016-07-24T14:41:58Z")

</div>

- Version: latest stable
- Operating System: Debian 8
- Steps to Reproduce:

## Description

There is 4 filebeat nodes, a 2 node kafka cluster, and data is being send from beat to kafka correctly in this structure.

```
fb fb fb .. fb
| | | |
   kf ------- kf

```

This issue is happened when one of the network between a fb node and a kf node is disabled(by setup a firewall and limiting the port visiting between two nodes), but network between kf nodes is ok, so the kf cluster is still in healthy status, like this.

```
fb fb fb .. fb fb fb fb .. fb
|X | | | ---------> | | |
  kf ------- kf kf ------- kf

```

Finally, no more data can be send from fb to kf cluster because of an endless reconnect attempt on fb.  
And This reconnect problem can be solved by kill the not connect kafka node from the cluster. Then data can be send again.

```
fb fb fb .. fb fb fb fb .. fb
   | | | ---------> | | | |
  kf ------- kf kf

```

## Assumption

From things happened above, i give my assumption:

- When a network disable happened, Strategy for choosing the next node is not suitable to discussing above!

- Which is caused by the source code shown below

- Since the kafka cluster is still in healthy status, metadata of the topic, leader of the replica, or the nodes have not changed!

- By the step of 1),2),3), It will still choose the same partion in the same node to send the data, And the endless loop will then happen

## Solution

Hopefully, It can be solve by taking a more thorough fault-tolerence strategy which takes different action to solve the different network errors

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 25, 2016, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-start-endless-reconnectiosn-when-connect-error-happen-to-kafka-cluster/56241/2 "2016-07-25T13:13:51Z")

</div>

see [my response here](https://discuss.elastic.co/t/bug-filebeat-cant-send-data-to-kafka-cluster-when-one-kafka-cant-connect-because-of-network/56184/2)

---

<div class="post-metadata">

**Author:** ![DeeeFOX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deeefox/32/49566_2.png) [@DeeeFOX](https://discuss.elastic.co/u/DeeeFOX)\
**Post date:** [July 26, 2016, 3:39am UTC](https://discuss.elastic.co/t/filebeat-start-endless-reconnectiosn-when-connect-error-happen-to-kafka-cluster/56241/3 "2016-07-26T03:39:45Z")

</div>

Oh thanks! And WangXiangUSTC is my work mate! What a little world!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2016, 2:42pm UTC](https://discuss.elastic.co/t/filebeat-start-endless-reconnectiosn-when-connect-error-happen-to-kafka-cluster/56241/4 "2016-08-14T14:42:12Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.