Testers wanted: Lumberjack integration with Hortonworks

Elastic Stack Logstash
#1

Hi there,

For those using logstash-forwarder but also using Hortonworks stack (HDP and HDF)

I have baked an experimental listener that allows Hortonworks Dataflow (a.k.a. Apache NiFi) to accept connections from logstash-forwarder agents.

The rationale for that is to allow for a drop in replacement for simpler pipelines (e.g. logstash-forwarder -> pipeline -> HDFS / Kafka / whatever) while providing stronger delivery guarantees than Logstash currently offers (NiFi's documentation provides insight over its data provenance and guarantees).

The code is still experimental and I would strongly appreciate if people could assist by testing it against a more diverse set of environments.

Cheers

#2

Interesting project!
However you know that this forwarder has been deprecated? We are moving users over to Filebeat these days.

#3

Mark,

Yes, I am aware of the deprecation, however its widespread use and the fact other applications make use of the protocol (e.g. node-lumberjack, python beaver, etc) means its use will continue for a while.

I plan to introduce support for libbeats once the protocol is formalised as its current implementation differs from its documentation

#4