Logstash processing only number of events 2 times to pipeline.max_inflight

Elastic Stack Logstash
1

My pipeline configuration as below

 - pipeline.id: staticmetrics
   queue.type: persisted
   pipeline.workers: 10
   pipeline.batch.size: 2
   pipeline.batch.delay: 1000
   path.config: "/Users/James/Desktop/metrics.conf"

Log says::

[staticmetrics] Starting pipeline {:pipeline_id=>"staticmetrics", "pipeline.workers"=>10, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>1000, "pipeline.max_inflight"=>20, "pipeline.sources"=>["/Users/James/Desktop/metrics.conf"], :thread=>"#<Thread:0x45762870 run>"}

Here the input has 600 records and expected 600 events in output. However we see only 40 entries present in output.

Here we can see only 40 events processed exactly 2 times the pipeline.max_inflight count. By changing the combination of pipeline.workers and pipeline.batch.size the outcome is same.

Any reason for this? any way we can let all incoming events processed?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.