My PR finally made it into master: https://github.com/elastic/beats/pull/684
There are 3 publisher options one can use:
Sync: If sync is used, PublishEvent and PublishEvents will block until we've got an ACK from elasticsearch or logstash.
- is longer wait times
- kinda disables load balancing if only on beat is running (still round-robin, but only one sink at once).
- recommended to use with some batches of events and PublishEvents only
- PublishEvents returns bool indicating success or failure
- (Dramatically) reduces memory usage as it limits number of event in flight
Guaranteed: If option is set, libbeat will retry publishing the events until success, no messages are dropped. Publish will only fail on shutdown, when pipes are cleared internally (to not halt shutdown due to unresponsive logstash/elasticsearch instance). This options is fully async. If queues in libbeat fill up, PublishEvent will block. If load-balancing is configured, there is no guarantee on indexing order in elasticsearch or logstash (Sync is required to fully guarantee order).
Signal: The option awaits any type implementing the outputs.Signaler interface. The Completed/Failed callbacks will be directly called from within the output plugin. To not block the publisher queue for too long, make sure these callbacks don't have much work to do.
Signal in some experimental filebeat fork. See publisher in filebeat guaranteeing success/failure of async batch publish events are reported in sequence to the registrar.
If you don't care about order when handling errors, there is no need for having a sorted list + flags. Instead you can have a Failed callback method forwarding failed batches (or just a batchID) on another channel to some worker handling fails.
I'm looking for common patterns to evolve for different use-cases, to provide helpers in libbeat for getting rid of some of the low-level handling. But we're not there yet.
In case your project is open-source I'm very interested to see the final result + learn about possible improvements to libbeat publisher.