# Pipeline.workers configuration and aggregation filter **URL:** \ **Category:** Logstash\ **Created:** [September 17, 2021, 8:27am UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461 "2021-09-17T08:27:21Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Roberto\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roberto_b/32/77615_2.png) [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Post date:** [September 17, 2021, 8:27am UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/1 "2021-09-17T08:27:21Z")
Hi all, I'm going to use a brand new server with 4 vCPU and 16GB RAM, I've some pipelines (+60) and I'll run multiple pipeline (for eg: 1 pipeline for 10 "easy pipelines" 8 for "medium pipelines" and so on). Some of these pipelines uses aggregation filter. I want to apply the best configuration in order to speed up the elaboration of the pipelines. If i set pipeline.workers to 2 each \*.conf will use 1 worker or the pipelines "shares" the 2 workers? what about the aggregation filter? I read about the fact that if I use an aggregation filter i must use only 1 worker per pipeline that use aggregation filter.It is correct? Kind Regards Roberto ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 17, 2021, 3:24pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/2 "2021-09-17T15:24:55Z")
> [@Roberto\_B](#): > > I read about the fact that if I use an aggregation filter i must use only 1 worker per pipeline that use aggregation filter.It is correct? Yes. In order to aggregate events those events must pass through the same instance of the aggregate filter, so you can only have one instance, which means one worker thread. If you have expensive processing before the aggregate (e.g. dns or geoip lookups, an elasticsearch, http or jdbc\_streaming filter) it is possible to use pipeline-to-pipeline communication to use multiple workers for the initial processing and then merge them into a single worker for the aggregate. Of course that will not retain the order of events. ---
**Author:** ![Roberto\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roberto_b/32/77615_2.png) [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Post date:** [September 17, 2021, 3:59pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/3 "2021-09-17T15:59:56Z")
> [@Roberto\_B](#): > > If i set pipeline.worke let me explain, if i run logstash specifiing to run this pipeline (with aggregation filter): `- pipeline.id: checksum path.config: "/etc/logstash/conf.d/checksum_*.conf" pipeline.workers: 2 ` Each conf file will take 1 worker? KR Roberto ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 17, 2021, 4:03pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/4 "2021-09-17T16:03:27Z")
No, logstash will combine all the files that match that regexp into a single configuration, and two worker threads will run that combined configuration. ---
**Author:** ![Roberto\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roberto_b/32/77615_2.png) [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Post date:** [September 17, 2021, 4:20pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/5 "2021-09-17T16:20:27Z")
And what if i have the following and run logstash without option (and without aggregation)? ```auto - pipeline.id: checksum_1 path.config: "/etc/logstash/conf.d/checksum_1.conf" pipeline.workers: 2 - pipeline.id: checksum_2 path.config: "/etc/logstash/conf.d/checksum_2.conf" pipeline.workers: 2 ``` logstash will use 2 worker per configuration (4 in total) ? KR Roberto ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 17, 2021, 4:28pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/6 "2021-09-17T16:28:55Z")
> [@Roberto\_B](#): > > logstash will use 2 worker per configuration (4 in total) ? Correct. ---
**Author:** ![Roberto\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roberto_b/32/77615_2.png) [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Post date:** [September 17, 2021, 4:53pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/7 "2021-09-17T16:53:23Z")
And just to be clear if i do the same with 1 worker per aggregation configuration specifying the name of the conf file i'll speed up the elaboration? ```auto - pipeline.id: checksum_1_aggr path.config: "/etc/logstash/conf.d/checksum_1.conf" pipeline.workers: 1 - pipeline.id: checksum_2_aggr path.config: "/etc/logstash/conf.d/checksum_2.conf" pipeline.workers: 1 - pipeline.id: checksum_3_aggr path.config: "/etc/logstash/conf.d/checksum_3.conf" pipeline.workers: 1 - pipeline.id: checksum_4_aggr path.config: "/etc/logstash/conf.d/checksum_4.conf" pipeline.workers: 1 ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 17, 2021, 6:20pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/8 "2021-09-17T18:20:39Z")
> [@Roberto\_B](#): > > i'll speed up the elaboration? I do not know what you mean by that. ---
**Author:** ![Roberto\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roberto_b/32/77615_2.png) [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Post date:** [September 17, 2021, 8:04pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/9 "2021-09-17T20:04:38Z")
Sorry, i forgotten to add the batch.size configuration. But now.it's clear, i must only configure 1 worker per aggregation pipeline and to increase the performance i can increase the Bach size of i correctly understood. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 15, 2021, 8:04pm UTC](https://discuss.elastic.co/t/pipeline-workers-configuration-and-aggregation-filter/284461/10 "2021-10-15T20:04:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.