Output beat events as plain HTTP POST


(Arnout Engelen) #1

The current suite of output modules doesn't include a publisher that simply POST's the events to some generic HTTP endpoint, without having to go through e.g. logstash.

I've made a PoC of this idea at https://github.com/elastic/beats/compare/master...raboof:httpOutput

Would such a thing be useful to include upstream?


Filebeat: Basic HTTP request output plugin
Http output for filebeat?
(Tudor Golubenco) #2

Thank you for opening this discussion, we do hear the request for a generic HTTP output from time to time.

In general, however, we're very conservative when it comes to adding new outputs. They tend to start simple, but require a lot of maintenance work over time, plus Logstash has a lot of outputs already, which can be used at the cost of adding another project. We've added Kafka & Redis to libbeat because these two are really popular as deployment options among our users, but at the same time planned to stop there :-).

In the particular case of HTTP, while it's great that the code change serves your needs, we can expect people to request bulking, custom parameters and headers, authentication mechanisms, different http versions, a.s.o as basic functionality. That's why we're quite reticent in accepting this patch.

Because we didn't have one before, I started an enhancement ticket to measure the interest in this, and we can reconsider if we see this being a must for many of our users.


(Arnout Engelen) #3

Agreed, I can see how this could be a slippery slope, and certainly slowly re-implementing most of Logstash as libbeat outputs is not really an appealing idea ;).

On the other hand requiring Logstash is a hurdle operationally: asking users to run the beat is one thing, but asking them to also configure and run a Logstash instance goes a bit further.

Makes sense. I wonder if we could allow additional outputs as 'third-party' extensions: outputs already register themselves on initialization, so that looks good, but I'm not sure I could register my own output in my beat without adding it to the list at libbeat/publisher/publish.go.

That might be a way to introduce a bit more flexibility, without taking on too much of a maintenance burden.

Thanks! You mention using the existing elasticsearch endpoint for this purpose. That might be an interesting avenue to explore, though I seem to remember considering it and deciding there was a bit too much ES-specific logic in there. Perhaps I should take another look ;).


(Steffen Siering) #4

Makes sense. I wonder if we could allow additional outputs as 'third-party' extensions: outputs already register themselves on initialization, so that looks good, but I'm not sure I could register my own output in my beat without adding it to the list at libbeat/publisher/publish.go.

That might be a way to introduce a bit more flexibility, without taking on too much of a maintenance burden.

Outputs already act kinda like plugins (just golang having no proper plugin support yet).

That is, it's already supported by beats. See this post in reply to very custom syslog output proposal: https://github.com/elastic/beats/pull/1525#issuecomment-217651768

The init function will be executed for any package being directly or indirectly imported by the main package.


(Arnout Engelen) #5

Interesting, I'll definitely check that out! Thanks for the pointer.


(Arnout Engelen) #6

Jup this seems to work out great! https://github.com/raboof/beats-output-http - thanks for the nudge :slight_smile:


(Steffen Siering) #7

Great it works. We hope this will keep maintenance somewhat managable (maintaining forks is always painful) until golang one day provides proper plugin support.


(system) #8

This topic was automatically closed after 21 days. New replies are no longer allowed.