Hi Team,
I am facing slowness issue in data sync after every restart of logstash. My observation is that input pipeline are taking time to start in parallel.
I am using Logstash 6.8 which is deployed on an EC2 instance(m5.4xlarge, 16vCPUs, RAM-64GB). I have allocated 15GB heap to Logstash.
I have around 8500 input pipelines running in every 30 second or in 1min. I am using mostly persistent queues to process data. We have configured persistent queue for different different entities in our application. We were running with these configuration since long but input pipelines are getting increased as we onboard new client. Below are the details for pipeline configuration.
| Entity Number | Input pipeline count | Persistent Queue | Persistent Queue Batch Size | Persistent Queue Worker Count |
|---|---|---|---|---|
| 1 | 177 | 1 | 500 | 1 |
| 2 | 1062 | 1 | 250 | 2 |
| 3 | 177 | 3 | 500 per pipeline | 1 worker per pipeline |
| 4 | 177 | 3 | 500 per pipeline | 1 worker per pipeline |
| 5 | 708 | 1 | 500 | 1 |
| 6 | 177 | 1 | 250 | 2 |
| 7 | 177 | 1 | 250 | 2 |
| 8 | 177 | 1 | 500 | 2 |
| 9 | 177 | 0 | 0 | 0 |
| 10 | 177 | 0 | 0 | 0 |
| 11 | 177 | 1 | 50 | 1 |
| 13 | 177 | 1 | 500 | 1 |
| 14 | 177 | 1 | 1000 | 2 |
| 15 | 177 | 1 | 500 | 3 |
| 16 | 177 | 1 | 500 | 2 |
| 17 | 177 | 0 | 0 | 0 |
| 18 | 177 | 1 | 500 | 2 |
| 19 | 177 | 1 | 1000 | 2 |
| 20 | 177 | 0 | 0 | 0 |
| 21 | 177 | 1 | 500 | 2 |
| 22 | 177 | 1 | 250 | 1 |
| 23 | 531 | 1 | 250 | 1 |
| 24 | 177 | 1 | 500 | 3 |
| 25 | 531 | 1 | 500 | 2 |
| 26 | 354 | 1 | 500 | 2 |
| 27 | 354 | 1 | 500 | 1 |
| 28 | 177 | 1 | 250 | 2 |
| 29 | 177 | 3 | 500 per pipeline | 1 worker per pipeline |
| 30 | 531 | 1 | 250 | 2 |
| 31 | 177 | 1 | 500 | 1 |
Below are few screenshots from monitoring tool which is showing increased number of blocked thread specially in [#]<jdbc_scheduler thread.
I have read the documents suggesting to increase number of persistent queues and divide the load on those persistent queues.
Will it help in my case? What kind of setup should I follow to support this as input pipelines will be increased day by day?
)