PV vs Metrics for benchmarking

Elastic Stack Logstash
1

Hello,

I am trying to load test my grok patterns and I am confused on the output of metrics{} vs using codec=> dots along with pv -War > /dev/null.

I am consuming the same 400,000 log lines as my input so that should be consistent and my output is as follows:

output{

  stdout {
    codec => dots
  }

  if "metrics" in [tags] {
    file {
      path => "/Users/jdyer/Projects/logstash/metics-%{+YYYY-MM-dd}.txt"
    }
  }
}

and when I run this I average ** [6.29KiB/s]** according to PV, which AFAIK means I am getting ~6000 events per second however metrics-2015-12-29 says I am getting much lower

{"@version":"1","@timestamp":"2015-12-29T04:30:00.328Z","message":"0.0.0.0","logstash_event":{"count":37,"rate_1m":0.0,"rate_5m":0.0,"rate_15m":0.0},"tags":["metrics"]}
{"@version":"1","@timestamp":"2015-12-29T04:30:05.293Z","message":"0.0.0.0","logstash_event":{"count":17267,"rate_1m":3452.2,"rate_5m":3452.2,"rate_15m":3452.2},"tags":["metrics"]}
{"@version":"1","@timestamp":"2015-12-29T04:30:10.289Z","message":"0.0.0.0","logstash_event":{"count":48634,"rate_1m":3452.2,"rate_5m":3452.2,"rate_15m":3452.2},"tags":["metrics"]}
{"@version":"1","@timestamp":"2015-12-29T04:30:15.289Z","message":"0.0.0.0","logstash_event":{"count":85654,"rate_1m":3678.0105642038648,"rate_5m":3498.8799201169877,"rate_15m":3467.8464972645706},"tags":["metrics"]}
{"@version":"1","@timestamp":"2015-12-29T04:30:20.288Z","message":"0.0.0.0","logstash_event":{"count":121775,"rate_1m":3975.988195096199,"rate_5m":3563.4391005264056,"rate_15m":3489.657818067379},"tags":["metrics"]}

You can see in those logs that the 1m rate is between 3400 and 3900 events per minute ( 56 to 65 events / second ).

Any idea whats up w/ the disparity, am I maybe misunderstanding how to benchmark my logstash config ?

Thanks !

-John

2

Anyone have any thoughts on this ?

3

I believe the rate metrics shown is the average number of events per second averaged over a certain time period, not the number of events over that time period. Having said that this does seem low given the total count provided and the frequency with which the events are generated (This seems to indicate closer to 6000 events per second as you suggest). What does your metrics filter configuration look like?

4

Thanks for responding, and happy new years !

So here is my metrics config, it's pretty straight forward

filter {
  metrics {
    meter => "logstash_event"
    rates => [ 1,5,15 ]
    clear_interval => -1
    add_tag => "metrics"
  }
}