I am trying to setup a geographically distributed EC clusters, where each DC has its own standalone EC cluster. Now I need to get the same messages from all the client system filebeat and topbeat to go to all the DCs so that all EC clusters get the same data. Kafka is the way to do it.
We badly need a Kafka output plugin for beats. It can solve all sorts of load balancing, resilience, throttling problems.
The current support for multiple logstash endpoint loadbalancing is not enough in this case.