# When will LogStash exceed the queue capacity and drop messages? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 25, 2014, 2:49pm UTC](https://discuss.elastic.co/t/when-will-logstash-exceed-the-queue-capacity-and-drop-messages/19437 "2014-08-25T14:49:15Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![shihpeng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shihpeng/32/40318_2.png) [@shihpeng](https://discuss.elastic.co/u/shihpeng)\ **Post date:** [August 25, 2014, 2:49pm UTC](https://discuss.elastic.co/t/when-will-logstash-exceed-the-queue-capacity-and-drop-messages/19437/1 "2014-08-25T14:49:15Z")
I am using LogStash to collect the logs from my service. The volume of the data is so large (20GB/day) that I am afraid that some of the data will be dropped at peak time. So I asked question [http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505](http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505) in Stack Overflow and decided to add a Redis as a buffer between ELB and LogStash to prevent data loss. However, I am curious about _when will LogStash exceed the queue capacity and drop messages?_ Because I've done some experiments and the result shows that LogStash can completely process all the data without any loss, e.g., local file (a 20GB text file) --\> LogStash --\> local file, netcat --\> LogStash --\> local file. Can someone give me a solid example (or scenario, if any) when LogStash eventually drops messages? So I can have a better understanding about why we need a buffer in front of it. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 25, 2014, 11:27pm UTC](https://discuss.elastic.co/t/when-will-logstash-exceed-the-queue-capacity-and-drop-messages/19437/2 "2014-08-25T23:27:58Z")
You should really ask this on the Logstash list - [https://groups.google.com/forum/#!forum/logstash-users](https://groups.google.com/forum/#!forum/logstash-users) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 26 August 2014 00:49, Shih-Peng Lin [shihpeng.lin@gmail.com](mailto:shihpeng.lin@gmail.com) wrote: > I am using LogStash to collect the logs from my service. The volume of > the data is so large (20GB/day) that I am afraid that some of the data will > be dropped at peak time. > > So I asked question > [http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505](http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505) in > Stack Overflow and decided to add a Redis as a buffer between ELB and > LogStash to prevent data loss. > > However, I am curious about _when will LogStash exceed the queue capacity > and drop messages?_ > > Because I've done some experiments and the result shows that LogStash can > completely process all the data without any loss, e.g., local file (a 20GB > text file) --\> LogStash --\> local file, netcat --\> LogStash --\> local file. > > Can someone give me a solid example (or scenario, if any) when LogStash > eventually drops messages? So I can have a better understanding about why > we need a buffer in front of it. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aJdffkGA135wQERfMjYdRZdTdDMXa11NHXOfvLHOpy9w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aJdffkGA135wQERfMjYdRZdTdDMXa11NHXOfvLHOpy9w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/when-will-logstash-exceed-the-queue-capacity-and-drop-messages/19437/3 "2017-07-06T01:06:35Z")