Hi there,
I have a pipeline that extracts documents from Elasticsearch and sends them to S3. I want to do some reconciliation on the process to prove that the number of documents extracted from Elasticsearch and the number of documents processed by the S3 output matches. Sometimes the upload fails or the customer suggests that the numbers 'appear' to be a little low.
The pipeline runs hourly and a simple count will solve the number of documents extracted part. By using the _node/stats/pipelines/<pipe_line>, I can work out the number of documents processed by the output filter.
The figure is cumulative which makes like for like hourly checks tricky.
All of the figures are reset on a pipeline reload, which is what is currently happening, after each reconciliation process finishes. The next time the check runs, it is like for like.
To achieve this, I have introduced a empty file into the Logstash configuration for that pipeline, that is modified by adding a space. This then forces a reload. As the file is empty, the functionality of the pipeline doesn't change.
Does anyone now whether there is a more graceful way to do this? I cannot see any API that allows this.
Thank you.