Heavy queuing on logstash nodes

Elastic Stack Logstash
1

Hello Team,

I have below architecture in place.

400-800 EC2                        ---> AWS NLB ---> 10 Logstash EC2s                      --->     14 EC2 of data node 
with filebeat                                                                                        i3.2xlarge (3 masters)   
                                                     100GB SSD with 500IOPS                                                     for queue.

Every day in evening we serve huge traffic which generates 80K events per/sec on logstash nodes for 30 mins, at the same time I can see elasticsearch with indexing rate of 150K docs/sec.

In this time filebeat sends data for 3 different indices along with 4th index being indexed by AWS lambda.
Out of these 3 index on which filebeat sending data, I need 1 index which should give me data for almost realtime.

Can someone suggest me how can I achieve this on this big scale ?
Daily ingestion is around 1.5 - 3 TB
For that 30 min window logstash consume SSD disk for queue with 11-15GB utilization on almost all logstash nodes.

2

Are you asking how to increase ingestion on Elasticsearch or Logstash?

3

I have doubt on logstash throughput along with elasticsearch ingestion.
First,
Why my group of logstash instances queuing so much of data ?

4

It will queue data if Elasticsearch cannot cope with the load.

5

But at the same time, I have few lambdas who ingest logs in elasticsarch, they are able to ingest data with 20 sec of lag means almost realtime, however logstash isn't.

6

can someone help here ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.