When will LogStash exceed the queue capacity and drop messages?

Elastic Stack Elasticsearch
1

I am using LogStash to collect the logs from my service. The volume of the
data is so large (20GB/day) that I am afraid that some of the data will be
dropped at peak time.

So I asked question
http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505 in
Stack Overflow and decided to add a Redis as a buffer between ELB and
LogStash to prevent data loss.

However, I am curious about when will LogStash exceed the queue capacity
and drop messages?

Because I've done some experiments and the result shows that LogStash can
completely process all the data without any loss, e.g., local file (a 20GB
text file) --> LogStash --> local file, netcat --> LogStash --> local file.

Can someone give me a solid example (or scenario, if any) when LogStash
eventually drops messages? So I can have a better understanding about why
we need a buffer in front of it.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

You should really ask this on the Logstash list -
https://groups.google.com/forum/#!forum/logstash-users

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 August 2014 00:49, Shih-Peng Lin shihpeng.lin@gmail.com wrote:

I am using LogStash to collect the logs from my service. The volume of
the data is so large (20GB/day) that I am afraid that some of the data will
be dropped at peak time.

So I asked question
http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505 in
Stack Overflow and decided to add a Redis as a buffer between ELB and
LogStash to prevent data loss.

However, I am curious about when will LogStash exceed the queue capacity
and drop messages?

Because I've done some experiments and the result shows that LogStash can
completely process all the data without any loss, e.g., local file (a 20GB
text file) --> LogStash --> local file, netcat --> LogStash --> local file.

Can someone give me a solid example (or scenario, if any) when LogStash
eventually drops messages? So I can have a better understanding about why
we need a buffer in front of it.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aJdffkGA135wQERfMjYdRZdTdDMXa11NHXOfvLHOpy9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.