Timing Questions with Logstash pipelines v. API Data

Elastic Stack Logstash
1

I have 9 logstash Pipelines that run to ingest data into ES.

I was trying to do some math initially by taking the time the pipeline starts ingest (Looking at the *.pipeline.log file looking at the time the query started and comparing it to the last document that was written into ES.

When we do that, we see about 3 - 5 minutes (assuming the network has some latency and other stuff going on). BUT when I look at the pipeline in the API i.e.
/stats/pipelines/PIPELINE_NAME?pretty and I see a duration_in_millis of like 7 seconds Which doesnt really line up.

So, what is the realistic way of calculating runtime of pipeline and validation that everything sent is ingested so I can report a complete time?

I did see this (_node/stats - duration_in_millis vs queue_push_duration_in_millis) which has some great info but doesnt really answer my question.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.