Pipeline stats API: in/out/filtered

monitoring

(James Beckett) #1

What do the in/out/filtered values in the events section of the pipeline stats API actually represent?

I had understood them to mean

  • in: total events entering the pipeline (sum of input plugins production)
  • out: total events successfully processed by output plugins
  • filtered: how many events made it through the filter stages (or possibly, how many were filtered out)

This doesn't seem to be the case - as far as I can see, all three values are identical. We have been recording these metrics to Graphite, and see no difference between in/out in any historical data. I was hoping to use this to flag any unexpected lossage in the pipelines.

Here's the stats output for a sample run using the generator plugin, showing the output plugin receiving zero events, but the out metric still showing 100:

"pipeline-metrics-test" : {
      "events" : {
        "duration_in_millis" : 62,
        "in" : 100,
        "out" : 100,
        "filtered" : 100,
        "queue_push_duration_in_millis" : 0
      },
      "plugins" : {
        "inputs" : [ {
          "id" : "random-generator-100",
          "events" : {
            "out" : 100,
            "queue_push_duration_in_millis" : 0
          },
          "name" : "generator"
        } ],
        "filters" : [ {
          "id" : "...",
          "events" : {
            "duration_in_millis" : 1,
            "in" : 100,
            "out" : 0
          },
          "name" : "drop"
        } ],
        "outputs" : [ {
          "id" : "...",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "stdout"
        } ]
      },

#2

I agree that these numbers do not represent what I expect them to represent.


(James Beckett) #3

Is this the same as https://github.com/elastic/logstash/issues/8752 ?


#4

Not certain, but I think so.


(system) closed #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.