Can Lumberjack plugin ensure 100% delivery guarantee?

[amruth]

Hi,

I am using http input and output plugins to transfer messages from one logstash to another logstash(this logstash send messages to Elasticsearch using elasticsearch plugin). I see nearly 20% of message loss and also first Logstash has some errors like

[HTTP Output Failure] Encountered non-2xx HTTP code 502 {:response_code=>502

I feel like there are issues with http plugin and wanted to migrate touse Lumberjack plugin. So can someone please say if Lumberjack delivery is 100% guaranteed?

Also is there any way we can know if Logstash is dropping messages(by saving them on local file)?

Thanks

[amruth]

Hi,

I've posted lot of questions related to http, tcp, udp input/output plugins, but I didn't get any detailed information on this.

I read somewhere that udp input plugin is not reliable, it helps a lot if you can maintain information on what is reliable and what is not.

Please shed some light on this.

Thanks in advance

[nikhil.k (Nikhil Kapoor)]

Hi @amruth

@amruth

I read somewhere that udp input plugin is not reliable, it helps a lot if you can maintain information on what is reliable and what is not.

I think non reliability of udp input plugin would mean as follows:-
Consider the scenario in which logstash/its hosts is terminated abnormally or if logstash is overloaded?

In this scenario, there would be data loss, as logstash input plugins (upd, tcp and many others) do not use request-response protocol. There would be no acknowledge receipt to the sender about the failure.

For eg. there are two machine(A & B).
machine A - sender
machine B - logstash

Machine A sends data to the machine B which is initially UP. Now suppose machine B gets down during data transfer. Here, sender would not be knowing the current status (down) of machine B and will continue to send the data, which arise to the problem of data loss.

The above limitation can be avoided by configuring persistent queue in logstash, which will provide the data durability in events of abnormal shutdown of Logstash or its host, ensuring at least once delivery.

In detail, please refer to the link:- Scaling Elasticsearch, Kibana, Beats, and Logstash | Elastic Blog

Regards
Nikhil Kapoor