How to check if pipeline execution completed in Logstash

Elastic Stack Logstash
1

We are using logstash 6.0 and in log it shows scheduled pipeline running, but how do I know if current run - input - filter - output has completed? Does following input JDBC INFO message with timespent is conclusion for the completion? Can this is be done using pipeline monitoring API? Like how many iterations, how much time spent etc per execution?

[2018-05-25T15:03:01,472][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"org_migration", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>375, :thread=>"#<Thread:0x3d3567ad@D:/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:290 run>"}
[2018-05-25T15:03:01,641][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"org_migration"}
[2018-05-25T15:03:01,657][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["org_migration"]}
[2018-05-25T15:05:05,604][INFO ][logstash.inputs.jdbc ] (0.454000s) select f1, f2 from tab
[2018-05-25T15:10:03,254][INFO ][logstash.inputs.jdbc ] (0.232000s) select f1, f2 from tab
[2018-05-25T15:15:02,778][INFO ][logstash.inputs.jdbc ] (0.232000s) select f1, f2 from tab

2

Well our observation - even if the the pipeline prints this log with time taken, still pipeline out put grows which means this time is not ideal to count pipeline time taken and not to conclude pipeline ended
Ref: https://www.elastic.co/guide/en/logstash/6.0/node-stats-api.html#pipeline-stats

But is there any (or upcoming) API to check if pipeline execution for a given iteration is over and metrics for the same?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.