Reachable_only: true setting with Winlogbeat Kafka output does not seem to work

[holdenkilbride]

Hello,

I am trying to set up winlogbeat to still publish to available partitions in the event the leader partition is unreachable (due to DNS, routing issues, etc.). The broker is still online in this scenario. I am just trying to ensure this can work in the event of an unreachable scenario. With the reachable_only setting set to true in my winlogbeat.yml config, it still appears as though the entire output gets blocked due to any partition becoming unreachable. Are there any other settings I might be missing, or ways to optimize my configuration to get this to do what I need? Thanks in advance.

This is using winlogbeat 6.4.2 and a kafka cluster running version 1.1.1

[steffens (Steffen Siering)]

The problem is, the partition still exists and the kafka cluster states the partition can be served. So the beat continues scheduling events for the partition. Eventually the queue will run full. The setting is active when a partition can not be served, because there is no active leader for said partition in the cluster. In a kafka cluster the leader for a partition can change at any time.

[holdenkilbride]

Assuming then that the leader will eventually change, what purpose does this setting serve?

[steffens (Steffen Siering)]

If you run kafka without replication and a broker goes down, then a partition becomes unavailable in the cluster. If the setting is enabled, the unavailable cluster will be ignored. It's the cluster state which indicates whether a partition is available or not.

In kafka each partition has it's own leader and followers if replication is enabled. Leaders and followers are kafka brokers. A partition is available as long as a leader can be found. If the leader broker goes down and another in-sync replica broker is availble, then one of the replicas eventually becomes the new leader.