Logstash Parallelism (pipeline workers) does not work with Persistent Queue

[zalseryani (Zeid ALSeryani)]

Issue

• We found that when setting pipeline.workers to 4 and having 4 CPU, filters and output were not running in parallel, while disabling the persistence queue and relying on memory , the parallel workers were working fine, knowing that I did not find this stated in logstash docs,

• Kindly advise If I am wrong.

Thank you.

[leandrojmp (Leandro Pereira)]

@zalseryani

We found that when setting pipeline.workers to 4 and having 4 CPU, filters and output were not running in parallel, while disabling the persistence queue and relying on memory , the parallel workers were working fine, knowing that I did not find this stated in logstash docs

Filters and output will use the specified number of workers regardless if you are using persisted queue or memory queue.

How did you find it was not working like this when using persisted queue? Please share any evidence of this.

[Rios]

Just to add to Leandro:

• Show LS log, would be nice in debug mode
• pipeline.workers - default Number of the host’s CPU cores, so no need to set to 4
• Have you checked LS github for existing issues?
• Which LS version do you use? Docker or on-premise?