Winlogbeat to Kafka : failover or loadbalancer

[blenski (Bruno Lenski)]

Hi

I am willing to set up winlogbeat to send logs to my Kafka cluster (3 machines)
I have seen that I can set several Kafka destination servers, thanks to the following config

output.kafka:

initial brokers for reading cluster metadata

hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]

My question is : is that a fall over set up or a load balancing one ?

If I want to balance the charge should I put a VIP in front of the Kakfa or use the above setting ?

Thanks for your help

Bruno Lenski

[bigphil (Philip Nunn)]

According to this filebeat document, the Kafka output handles loadbalancing internally. I can only assume that Winlogbeat uses the same Kafka output as Filebeat.

[steffens (Steffen Siering)]

When using kafka it is the client that is responsible for load balancing. It's a load-balance always service! Clients have to choose a partition and then send events to the current leader of that partition. Do not put kafka behind HAProxy or any other load balancer. The kafka leader provides a cluster state with a (topic/partition ID)->kafka host mapping to the client.