Logstash Parallelism (pipeline workers) does not work with Persistent Queue

Elastic Stack Logstash
1

Issue

  • We found that when setting pipeline.workers to 4 and having 4 CPU, filters and output were not running in parallel, while disabling the persistence queue and relying on memory , the parallel workers were working fine, knowing that I did not find this stated in logstash docs,

  • Kindly advise If I am wrong.

Thank you.

2

Filters and output will use the specified number of workers regardless if you are using persisted queue or memory queue.

How did you find it was not working like this when using persisted queue? Please share any evidence of this.

3

Just to add to Leandro:

  • Show LS log, would be nice in debug mode
  • pipeline.workers - default Number of the host’s CPU cores, so no need to set to 4
  • Have you checked LS github for existing issues?
  • Which LS version do you use? Docker or on-premise?
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.