About a year and a half ago we added a number of options to a local fork of logstash-forwarder, for high availability.
The two main things we needed at the time were:
- Having a concept of "destinations". Each input would be tagged with a specific destination to send to. Multiple destinations (ie, multiple ELK clusters) could be configured as outputs, and each would take the log lines given to it and forward them to the right place.
- Connecting to all listed output servers simultaneously, and sending lines to all of them at the same time. This way, if one server goes down, we don't block the entire pipeline trying to send logs to it. It also enables us to get much higher throughput. In this case, all logstash servers use roughly equal amounts of CPU. If one server gets too slow, it ends up getting less traffic until it recovers.
We're going to migrate to Filebeat soon, and would very much like to contribute these changes back upstream.
If anyone has thoughts on how these changes should look, specific things we'd need to address in order to get these accepted, etc, we'd love to hear them.