I am using http input and output plugins to transfer messages from one logstash to another logstash(this logstash send messages to Elasticsearch using elasticsearch plugin). I see nearly 20% of message loss and also first Logstash has some errors like
I feel like there are issues with http plugin and wanted to migrate touse Lumberjack plugin. So can someone please say if Lumberjack delivery is 100% guaranteed?
Also is there any way we can know if Logstash is dropping messages(by saving them on local file)?
I think non reliability of udp input plugin would mean as follows:-
Consider the scenario in which logstash/its hosts is terminated abnormally or if logstash is overloaded?
In this scenario, there would be data loss, as logstash input plugins (upd, tcp and many others) do not use request-response protocol. There would be no acknowledge receipt to the sender about the failure.
For eg. there are two machine(A & B).
machine A - sender
machine B - logstash
Machine A sends data to the machine B which is initially UP. Now suppose machine B gets down during data transfer. Here, sender would not be knowing the current status (down) of machine B and will continue to send the data, which arise to the problem of data loss.
The above limitation can be avoided by configuring persistent queue in logstash, which will provide the data durability in events of abnormal shutdown of Logstash or its host, ensuring at least once delivery.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.