We are not able to get Kafka to failover correctly when our first Kafka broker is stopped. We want beats data to flow to our other Kafka brokers in our cluster. We have a 3 node broker cluster. Our first IP address is the one ending in .160 and that is the one that makes or breaks the pipeline when stopped.
I verified that clustering is working correctly. I created a test topic within kafka and ran the kafka-console-producer.sh to inject data. Then I ran the kafka-console-consumer.sh to make sure the same data was showing up on each kafka node. I stopped the .160 broker and data continued to work in my simulation. This leads me to believe the issue is in our beats config or maybe old configured information.
We are just now configuring the other 2 nodes in the Kafka output in winlogbeat, before it was only pointed to .160. I have tried creating a test topic generated from winlogbeat just to make sure the winlogbeat topic we currently have isn't the problem. However, when I stop the kafka service on that node, data does not continue to flow to from winlogbeats to the other broker IPs are in the .yml. I was running the kafka-console-consumer.sh again to follow this.
Just need some direction in our beats config. I am not sure if add the other two nodes to the winlogbeat.yml is what is causing an issue or something else we are missing. Any help would be greatly appreciated.
On a side note we also send filebeats data to the same Kafka broker cluster. I don't know if the lack of a "client" parameter is causing issues. I know winlogbeat and filebeat will set the client in kafka automatically to "beats" if not specified. Not sure if they are both setting it to the same thing and if that is causing issues.
Here is out kafka output in winlogbeat.yml:
output.kafka:
initial brokers for reading cluster metadata
hosts: ["10.0.0.160:9092", "10.0.0.161:9092", "10.0.0.162:9092"]
message topic selection + partitioning
#topic: '%{[type]}'
topic: 'winlogbeat'
partition.round_robin:
reachable_only: false
required_acks: 1
#compression: gzip
#max_message_bytes: 1000000