I have 9 logstash Pipelines that run to ingest data into ES.
I was trying to do some math initially by taking the time the pipeline starts ingest (Looking at the *.pipeline.log file looking at the time the query started and comparing it to the last document that was written into ES.
When we do that, we see about 3 - 5 minutes (assuming the network has some latency and other stuff going on). BUT when I look at the pipeline in the API i.e.
/stats/pipelines/PIPELINE_NAME?pretty and I see a duration_in_millis of like 7 seconds Which doesnt really line up.
So, what is the realistic way of calculating runtime of pipeline and validation that everything sent is ingested so I can report a complete time?
I did see this (_node/stats - duration_in_millis vs queue_push_duration_in_millis) which has some great info but doesnt really answer my question.