Multiple destinations for failover

[Mazhar]

I am configuring the following at my end,

output.kafka:
enabled:true
hosts:["kafkahost1:port1","kafkahost2:port2"]

topic: topic_name

From the configuration I understand I am sending the same logs/event to multiple receivers, but I am very eager to know how do I configure fail-over destinations. For example if my kafkahost1:port1 is not active, the log/event has to reach kafkahost2:port2.

I did not find much details on the forum, hence adding my points/queries here.

Regards,
Mazhar

[Mazhar]

@andrewkroh - Can you help me here.

//Mazhar

[andrewkroh]

hosts specifies the addresses of the Kafka brokers where cluster metadata will be fetched. If one is unreachable it will fail-over to the next.

It will use the cluster metadata to determine the partitions for the configured topic. Then will use the metadata to determine the leader for each of the partitions. Then it will start sending data to the host/port associated with the partition leader.

If a partition goes down it can block the output. But you can set reachable_only: true to have it ignore that partition and move on. See https://www.elastic.co/guide/en/beats/winlogbeat/current/kafka-output.html

[Mazhar]

Thanks. I am not fully understanding the below parameters, can you put some light on it

topic: '%{[fields.log_topic]}'
partition.round_robin:

[andrewkroh]

@Mazhar

topic: '%{[fields.log_topic]}'

This selects the topic based on the fields.log_topic value from the event being published. It allows for the topic to be selected dynamically based on the event contents.

@Mazhar

partition.round_robin:

This is part of the configuration for choosing round robin as the partitioning method. It will send a batch of events to one partition then move on to another partition.