@ruflin I know this is a slightly older thread, although I do have a use case were I'd really like to see HTTP output for libbeat. With our current multi-tentant setup, all servers ship logs with logstash-forwarder to a central Logstash receiving servers (multiple, behind a load blancer), which then outputs to a Redis list for processing at a later time.
The issue we're having is that logstash-forwarder opens up persistent TCP connections, which ends up being a pain when you're connecting behind a load balancer, even if you tweak the parameters accordingly. I understand that the lubmerjack protocol is potentially more efficient and well built for shipping events, but it doesn't scale as easily and as well as simple HTTP would. If libbeat had the option to send batches of events over HTTP, these requests would be shorter lived and easier to balance more evenly across multiple logstash servers using HTTP input or some other custom lightweight HTTP server which could send of the data off to logstash. The other thing to note is that HTTP is quite standard so it becomes easy at that point for any application to send off an event to a Logstash endpoint, without even requiring the filebeat shipper in some cases. Now you might be wondering why we don't just install Logstash on each machine? That's because we have many servers that have very limited resources and we'd also rather have a small lightweight process with a small memory footprint, compared to a large Java process with unnecessary complexity.
I am aware that there would be a cost to using HTTP, such as continuously opening new connections, http protocol overhead (larger message size), etc, but in my situation (and i'm sure others would agree), the benefits of an http output still outweigh the cons. Now considering all of this, would the beats team consider adding an HTTP output module?