I am using LogStash to collect the logs from my service. The volume of the
data is so large (20GB/day) that I am afraid that some of the data will be
dropped at peak time.
So I asked question
Stack Overflow and decided to add a Redis as a buffer between ELB and
LogStash to prevent data loss.
However, I am curious about when will LogStash exceed the queue capacity
and drop messages?
Because I've done some experiments and the result shows that LogStash can
completely process all the data without any loss, e.g., local file (a 20GB
text file) --> LogStash --> local file, netcat --> LogStash --> local file.
Can someone give me a solid example (or scenario, if any) when LogStash
eventually drops messages? So I can have a better understanding about why
we need a buffer in front of it.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.