# Heavy queuing on logstash nodes **URL:** \ **Category:** Logstash\ **Created:** [September 26, 2020, 10:01am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050 "2020-09-26T10:01:38Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![suyog1](https://avatars.discourse-cdn.com/v4/letter/s/f04885/32.png) [@suyog1](https://discuss.elastic.co/u/suyog1)\ **Post date:** [September 26, 2020, 10:01am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/1 "2020-09-26T10:01:39Z")
Hello Team, I have below architecture in place. ``` 400-800 EC2 ---> AWS NLB ---> 10 Logstash EC2s ---> 14 EC2 of data node with filebeat i3.2xlarge (3 masters) 100GB SSD with 500IOPS for queue. ``` Every day in evening we serve huge traffic which generates 80K events per/sec on logstash nodes for 30 mins, at the same time I can see elasticsearch with indexing rate of 150K docs/sec. In this time filebeat sends data for 3 different indices along with 4th index being indexed by AWS lambda. Out of these 3 index on which filebeat sending data, I need 1 index which should give me data for almost realtime. Can someone suggest me how can I achieve this on this big scale ? Daily ingestion is around 1.5 - 3 TB For that 30 min window logstash consume SSD disk for queue with 11-15GB utilization on almost all logstash nodes. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 28, 2020, 6:04am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/2 "2020-09-28T06:04:55Z")
Are you asking how to increase ingestion on Elasticsearch or Logstash? ---
**Author:** ![suyog1](https://avatars.discourse-cdn.com/v4/letter/s/f04885/32.png) [@suyog1](https://discuss.elastic.co/u/suyog1)\ **Post date:** [September 28, 2020, 6:19am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/3 "2020-09-28T06:19:34Z")
I have doubt on logstash throughput along with elasticsearch ingestion. First, Why my group of logstash instances queuing so much of data ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 28, 2020, 6:21am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/4 "2020-09-28T06:21:12Z")
It will queue data if Elasticsearch cannot cope with the load. ---
**Author:** ![suyog1](https://avatars.discourse-cdn.com/v4/letter/s/f04885/32.png) [@suyog1](https://discuss.elastic.co/u/suyog1)\ **Post date:** [September 28, 2020, 6:23am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/5 "2020-09-28T06:23:26Z")
But at the same time, I have few lambdas who ingest logs in elasticsarch, they are able to ingest data with 20 sec of lag means almost realtime, however logstash isn't. ---
**Author:** ![suyog1](https://avatars.discourse-cdn.com/v4/letter/s/f04885/32.png) [@suyog1](https://discuss.elastic.co/u/suyog1)\ **Post date:** [September 28, 2020, 7:20am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/6 "2020-09-28T07:20:47Z")
can someone help here ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2020, 7:20am UTC](https://discuss.elastic.co/t/heavy-queuing-on-logstash-nodes/250050/7 "2020-10-26T07:20:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.