# Logstash Parallelism (pipeline workers) does not work with Persistent Queue

**URL:** <https://discuss.elastic.co/t/logstash-parallelism-pipeline-workers-does-not-work-with-persistent-queue/342673>\
**Category:** Logstash\
**Created:** [September 10, 2023, 8:17am UTC](https://discuss.elastic.co/t/logstash-parallelism-pipeline-workers-does-not-work-with-persistent-queue/342673 "2023-09-10T08:17:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![zalseryani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zalseryani/32/122438_2.png) [@zalseryani](https://discuss.elastic.co/u/zalseryani)\
**Post date:** [September 10, 2023, 8:17am UTC](https://discuss.elastic.co/t/logstash-parallelism-pipeline-workers-does-not-work-with-persistent-queue/342673/1 "2023-09-10T08:17:15Z")

</div>

# Issue

- We found that when setting pipeline.workers to 4 and having 4 CPU, filters and output were not running in parallel, while disabling the persistence queue and relying on memory , the parallel workers were working fine, knowing that I did not find this stated in logstash docs,

- Kindly advise If I am wrong.

Thank you.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 10, 2023, 12:41pm UTC](https://discuss.elastic.co/t/logstash-parallelism-pipeline-workers-does-not-work-with-persistent-queue/342673/2 "2023-09-10T12:41:06Z")

</div>

> [@zalseryani](#):
>
> We found that when setting pipeline.workers to 4 and having 4 CPU, filters and output were not running in parallel, while disabling the persistence queue and relying on memory , the parallel workers were working fine, knowing that I did not find this stated in logstash docs

Filters and output will use the specified number of workers regardless if you are using persisted queue or memory queue.

How did you find it was not working like this when using persisted queue? Please share any evidence of this.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 11, 2023, 6:44am UTC](https://discuss.elastic.co/t/logstash-parallelism-pipeline-workers-does-not-work-with-persistent-queue/342673/3 "2023-09-11T06:44:15Z")

</div>

Just to add to Leandro:

- Show LS log, would be nice in debug mode
- pipeline.workers - default `Number of the host’s CPU cores`, so no need to set to 4
- Have you checked [LS github](https://github.com/elastic/logstash/issues) for existing issues?
- Which LS version do you use? Docker or on-premise?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2023, 6:45am UTC](https://discuss.elastic.co/t/logstash-parallelism-pipeline-workers-does-not-work-with-persistent-queue/342673/4 "2023-10-09T06:45:06Z")

</div>

Closed on October 9, 2023, 6:45am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.