What do the in/out/filtered values in the events section of the pipeline stats API actually represent?
I had understood them to mean
-
in: total events entering the pipeline (sum of input plugins production) -
out: total events successfully processed by output plugins -
filtered: how many events made it through the filter stages (or possibly, how many were filtered out)
This doesn't seem to be the case - as far as I can see, all three values are identical. We have been recording these metrics to Graphite, and see no difference between in/out in any historical data. I was hoping to use this to flag any unexpected lossage in the pipelines.
Here's the stats output for a sample run using the generator plugin, showing the output plugin receiving zero events, but the out metric still showing 100:
"pipeline-metrics-test" : {
"events" : {
"duration_in_millis" : 62,
"in" : 100,
"out" : 100,
"filtered" : 100,
"queue_push_duration_in_millis" : 0
},
"plugins" : {
"inputs" : [ {
"id" : "random-generator-100",
"events" : {
"out" : 100,
"queue_push_duration_in_millis" : 0
},
"name" : "generator"
} ],
"filters" : [ {
"id" : "...",
"events" : {
"duration_in_millis" : 1,
"in" : 100,
"out" : 0
},
"name" : "drop"
} ],
"outputs" : [ {
"id" : "...",
"events" : {
"duration_in_millis" : 0,
"in" : 0,
"out" : 0
},
"name" : "stdout"
} ]
},