The current suite of output modules doesn't include a publisher that simply POST's the events to some generic HTTP endpoint, without having to go through e.g. logstash.
I've made a PoC of this idea at https://github.com/elastic/beats/compare/master...raboof:httpOutput
Would such a thing be useful to include upstream?
Thank you for opening this discussion, we do hear the request for a generic HTTP output from time to time.
In general, however, we're very conservative when it comes to adding new outputs. They tend to start simple, but require a lot of maintenance work over time, plus Logstash has a lot of outputs already, which can be used at the cost of adding another project. We've added Kafka & Redis to libbeat because these two are really popular as deployment options among our users, but at the same time planned to stop there :-).
In the particular case of HTTP, while it's great that the code change serves your needs, we can expect people to request bulking, custom parameters and headers, authentication mechanisms, different http versions, a.s.o as basic functionality. That's why we're quite reticent in accepting this patch.
Because we didn't have one before, I started an enhancement ticket to measure the interest in this, and we can reconsider if we see this being a must for many of our users.
Agreed, I can see how this could be a slippery slope, and certainly slowly re-implementing most of Logstash as libbeat outputs is not really an appealing idea ;).
On the other hand requiring Logstash is a hurdle operationally: asking users to run the beat is one thing, but asking them to also configure and run a Logstash instance goes a bit further.
Makes sense. I wonder if we could allow additional outputs as 'third-party' extensions: outputs already register themselves on initialization, so that looks good, but I'm not sure I could register my own output in my beat without adding it to the list at libbeat/publisher/publish.go.
That might be a way to introduce a bit more flexibility, without taking on too much of a maintenance burden.
Thanks! You mention using the existing elasticsearch endpoint for this purpose. That might be an interesting avenue to explore, though I seem to remember considering it and deciding there was a bit too much ES-specific logic in there. Perhaps I should take another look ;).
Makes sense. I wonder if we could allow additional outputs as 'third-party' extensions: outputs already register themselves on initialization, so that looks good, but I'm not sure I could register my own output in my beat without adding it to the list at libbeat/publisher/publish.go.
That might be a way to introduce a bit more flexibility, without taking on too much of a maintenance burden.
Outputs already act kinda like plugins (just golang having no proper plugin support yet).
That is, it's already supported by beats. See this post in reply to very custom syslog output proposal: New syslog output by avleen · Pull Request #1525 · elastic/beats · GitHub
The init function will be executed for any package being directly or indirectly imported by the main package.
Interesting, I'll definitely check that out! Thanks for the pointer.
Jup this seems to work out great! GitHub - raboof/beats-output-http: HTTP output producer for the Elastic Beats framework - thanks for the nudge 
Great it works. We hope this will keep maintenance somewhat managable (maintaining forks is always painful) until golang one day provides proper plugin support.
This topic was automatically closed after 21 days. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.