Logstash processing only number of events 2 times to pipeline.max_inflight

My pipeline configuration as below

 - pipeline.id: staticmetrics
   queue.type: persisted
   pipeline.workers: 10
   pipeline.batch.size: 2
   pipeline.batch.delay: 1000
   path.config: "/Users/James/Desktop/metrics.conf"

Log says::

[staticmetrics] Starting pipeline {:pipeline_id=>"staticmetrics", "pipeline.workers"=>10, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>1000, "pipeline.max_inflight"=>20, "pipeline.sources"=>["/Users/James/Desktop/metrics.conf"], :thread=>"#<Thread:0x45762870 run>"}

Here the input has 600 records and expected 600 events in output. However we see only 40 entries present in output.

Here we can see only 40 events processed exactly 2 times the pipeline.max_inflight count. By changing the combination of pipeline.workers and pipeline.batch.size the outcome is same.

Any reason for this? any way we can let all incoming events processed?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.