# Starting Point to Tune Logstash **URL:** \ **Category:** Logstash\ **Created:** [June 19, 2025, 5:01am UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301 "2025-06-19T05:01:24Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [June 19, 2025, 5:01am UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/1 "2025-06-19T05:01:24Z")
Hi there i have a dedicated logstash machine with 16 cores and 32 Gb RAM and multiple pipelines inside it. There are 3 heavy pipelines(complex filters and high event rate). what is a good starting point to make a global config in logstash.yml for each pipeline and a custom config for those 3 heavy pipelines in pipelines.yml? The total pipeline in my machine is 26 (including those 3) Should I give the default value (16 and 125) for pipeline.workers and pipeline.batch.size in logstash.yml? Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 19, 2025, 4:05pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/2 "2025-06-19T16:05:15Z")
If you have a pipeline worker thread with work to do then it needs a CPU to execute on. If you have 26 pipelines, each with multiple worker threads, then as the system runs out of CPU it will spend more and more time thread switching. As a result it will not scale linearly as load increases. My advice is to minimize the number of workers threads (possibly as low as 1 per pipeline). If you feel some pipelines need additional workers then add them and _measure_ the system at high load to see whether the additional workers cause an increase or decrease in throughput. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [June 20, 2025, 6:04am UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/3 "2025-06-20T06:04:55Z")
Noted, thank you But do you think I should also adjust the pipeline.batch.size or keep it default for the starting point? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 20, 2025, 12:15pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/4 "2025-06-20T12:15:20Z")
I would start with the default value. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [June 20, 2025, 12:35pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/5 "2025-06-20T12:35:30Z")
ok, thank you so much. i would start tuning with that config ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 20, 2025, 12:59pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/6 "2025-06-20T12:59:29Z")
Also you should monitor LS [statistics](https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodestatspipelines). You can see how events will be processed if you change parameter/s. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [July 2, 2025, 1:11pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/7 "2025-07-02T13:11:33Z")
hi @Rios i followed your suggestion and I found that logstash took a long time to publish an event to Elastic. As you can see here, I separate the output config for the index that will be monthly, weekly, or daily indices: ```auto "outputs" : [ { "id" : "ocp-daily", "documents" : { "dlq_routed" : 1216, "successes" : 674948 }, "bulk_requests" : { "with_errors" : 1163, "successes" : 75063, "responses" : { "200" : 76226 } }, "events" : { "duration_in_millis" : 3444186, "in" : 676164, "out" : 676164 }, "name" : "elasticsearch", "flow" : { "worker_millis_per_event" : { "current" : 6.683, "last_1_minute" : 5.304, "last_5_minutes" : 5.224, "last_15_minutes" : 5.587, "last_1_hour" : 5.334, "lifetime" : 5.094 }, "worker_utilization" : { "current" : 7.488, "last_1_minute" : 7.563, "last_5_minutes" : 6.668, "last_15_minutes" : 6.895, "last_1_hour" : 7.046, "lifetime" : 6.96 } } ``` ```auto { "id" : "ocp-monthly", "documents" : { "dlq_routed" : 357, "successes" : 223974 }, "bulk_requests" : { "with_errors" : 268, "successes" : 62649, "responses" : { "200" : 62917 } }, "events" : { "duration_in_millis" : 2218478, "in" : 224331, "out" : 224331 }, "name" : "elasticsearch", "flow" : { "worker_millis_per_event" : { "current" : 10.32, "last_1_minute" : 10.4, "last_5_minutes" : 9.649, "last_15_minutes" : 9.987, "last_1_hour" : 10.21, "lifetime" : 9.889 }, "worker_utilization" : { "current" : 4.293, "last_1_minute" : 4.5, "last_5_minutes" : 4.232, "last_15_minutes" : 4.446, "last_1_hour" : 4.509, "lifetime" : 4.483 } } } ``` I don't understand why logstash takes so long to publish an event. you can see that logstash needs 10ms to publish each event. my configuration was simple. What else can I tune from this? ```auto if [merge] == "month" { elasticsearch { id => "ocp-monthly" hosts => ["https://host1:9215", "https://host2:9215", and so on] ssl_certificate_authorities => '/logstash/logstash-8.18.2/config/certs/ca.crt' ssl_verification_mode => 'none' index => "ocp-sby-%{[kubernetes][namespace]}-%{+YYYY.MM}" document_id => "%{[custom_id]}" user => '${ES_USER}' password => '${ES_PWD}' } } else if [merge] == "weekly" { elasticsearch { hosts => ["https://host1:9215", "https://host2:9215", and so on"] ssl_certificate_authorities => '/logstash/logstash-8.18.2/config/certs/ca.crt' ssl_verification_mode => 'none' index => "ocp-sby-%{[kubernetes][namespace]}-%{+xxxx.ww}" document_id => "%{[custom_id]}" user => '${ES_USER}' password => '${ES_PWD}' } } else { elasticsearch { id => "ocp-daily" hosts => ["https://host1:9215", "https://host2:9215", and so on"] ssl_certificate_authorities => '/logstash/logstash-8.18.2/config/certs/ca.crt' ssl_verification_mode => 'none' index => "ocp-sby-%{[kubernetes][namespace]}-%{+YYYY.MM.dd}" document_id => "%{[custom_id]}" user => '${ES_USER}' password => '${ES_PWD}' } } ``` I took a quick look at the monitoring cluster, and it turns out that almost all pipelines have been publishing events for a long time to Elastic My elastic cluster consists of 7 Data nodes with 16GB of heap each. Thanks ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 3, 2025, 2:41pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/8 "2025-07-03T14:41:38Z")
It's not easy to get details from your posts. To summarize: - 23 pipelines + 3 under heavy loads on 16 cores&32 GB memory - ~10ms is event processing time for heavy loads - everything is on 1 LS host - pipeline.workers and pipeline.batch.size are default? ```auto pipeline.workers # LS will use max 16 pipeline.batch.size: 125 pipeline.batch.delay: 5 pipeline.ordered: auto ``` - What are your values Xms and Xmx in jvm.options? - Are all settings the same for pipelines? What is specific for those 3 pipelines? - Are you using memory or persistent queue? - How many data ES nodes are using? - Have you checked ES logs? Especially because slow insert and DLQ. - What is avg/max the message in those h loaded pipelines? Since you said "my configuration was simple", there is not much code in filters, you can try with: _Edit: make backup before any changes_ ```auto pipeline.batch.size: 250 # only in heavy loads compression_level => 5 # or increase to reduce load ssl_enabled => true # yes, should be by default pool_max => 2000 # increase to reduce reopening pool_max_per_route => 200 ssl_supported_protocols => "TLSv1.3" # use only 1.3, should be faster to establish sec channel resurrect_delay => 2 ``` - Exclude dedicated master nodes from list - Check ES logs(all nodes), why are you getting dlq\_routed, you can do it manually or metricbeat or agent - Use sniffing mode, check this [thread](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/4). - Investigate LS statistics for all pipelines - Check value for `tcp_keepalive_time`, only check, do not touch on OS level. - Allocate 2-3 nodes only to heavy loaded pipelines, other pipelines should This is not simple optimization activity since it's on live data&load, where Jedi council don't have full information or access. I truly hope other Jedi will give own opinion. Have you used live [pipelines monitoring](https://www.elastic.co/docs/reference/logstash/logstash-monitoring-ui) in Kibana? If not already exists, should set it ```auto PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.enabled": true } } ``` ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [July 3, 2025, 6:13pm UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/9 "2025-07-03T18:13:13Z")
I don't know if it is relevant but I had to change open file limit on linux for heavy load pipeline. cat /usr/lib/systemd/system/logstash.service |grep NOFILE LimitNOFILE=66384 default is 16k ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [July 4, 2025, 7:10am UTC](https://discuss.elastic.co/t/starting-point-to-tune-logstash/379301/10 "2025-07-04T07:10:25Z")
> [@Rios](#): > > everything is on 1 LS host I have 2 logstash server > [@Rios](#): > > pipeline.workers and pipeline.batch.size are default? This configuration is in logstash.yml workers = 8 batch size = 125 delay = default in pipelines.yml, I made a custom config for those heavy pipelines ```auto - pipeline.id: api path.config: /logstash/logstash-8.18.2/pipelines/api.conf pipeline.workers: 10 pipeline.batch.size: 256 queue.type: persisted queue.max_bytes: 8gb - pipeline.id: ocp-sby path.config: /logstash/logstash-8.18.2/pipelines/ocp-sby.conf pipeline.workers: 10 pipeline.batch.size: 256 - pipeline.id: ocp-jkt path.config: /logstash/logstash-8.18.2/pipelines/ocp-jkt.conf pipeline.workers: 10 pipeline.batch.size: 256 ``` > [@Rios](#): > > What are your values Xms and Xmx in jvm.options? 16GB > [@Rios](#): > > How many data ES nodes are using? I put all those 7 data nodes in all the pipelines. Of course, 3 of them were master-eligible nodes too. I don't have dedicated master nodes. > [@Rios](#): > > Have you checked ES logs? Especially because slow insert and DLQ. The log seems fine. From one of the master-eligible nodes, I don't see anything that causes Elastic to slow down from doing its job, such as frequent GC or something similar. Most of it was an error mapping log, and I don't activate DLQ either > [@Rios](#): > > What is avg/max the message in those h loaded pipelines? From the monitoring cluster, I can see: For the API pipeline, around 7k events per second For OCP pipelines, around 3k events per second > [@Rios](#): > > Exclude dedicated master nodes from list Unfortunately, I don't have it in my cluster, and I may try the pool configuration first to see if it has an effect. thanks > [@Rios](#): > > Have you used live [pipelines monitoring](https://www.elastic.co/docs/reference/logstash/logstash-monitoring-ui) in Kibana? If not already exists, should set it Don't worry, my monitoring cluster already has it