Logstash-filter-metrics used 100% CPU?

[srdegeus]

I wanted to get an idication of the ingest speed of logstash and found that the (buildin) plugin logstash-filter-metrics could do this.

After adding the example code to my config file, logstash starts printing a rate of 20000+ and consuming 100% CPU

I tried with just the example code, but this has the same behaviour. It starts using 100% CPU (on all 4 CPUs)

Is this expected behaviour? Am I missing something?

I did not change any config except setting JAVA_HOME to C:\Program Files\Java\jre1.8.0_121

I am running this on Windows Server 2016 (4 CPU, 4GB mem) :
D:\ELK\logstash>bin\logstash.bat -V
Could not find log4j2 configuration at path /ELK/logstash/config/log4j2.properties. Using default config which logs to console
logstash 5.2.2
jruby 1.7.25 (1.9.3p551) 2016-04-13 867cb81 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 +jit [Windows NT (unknown)-amd64]
java 1.8.0_121 (Oracle Corporation)
jvm Java HotSpot(TM) 64-Bit Server VM / 25.121-b13

[Christian_Dahlqvist]

If there is nothing upstream or downstream that is limiting throughput, Logstash performance is generally limited by the amount of CPU available. Assuming your simple test has no inputs or outputs affecting performance, it is natural for Logstash to saturate the CPUs of the host at maximum throughput.

[srdegeus]

I used the simple test as it is on the logstash-filter-metrics page

input {
  generator {
    type => "generated"
  }
}

filter {
  if [type] == "generated" {
    metrics {
      meter => "events"
      add_tag => "metric"
    }
  }
}

output {
  # only emit events with the 'metric' tag
  if "metric" in [tags] {
    stdout {
      codec => line {
        format => "rate: %{[events][rate_1m]}"
      }
    }
  }
}

What I didn't realise is that the

input {
  generator {
    type => "generated"
  }
}

Generates random "events" (as fast as it can go) and that those are being measured.

It took a bit of trial and error to get my throughput measured as it doesn't seem to work on the default "message" field.
I think the following would be a nice example as this doesn't use 100% CPU, but measures how fast you press the enter key:

input {
  stdin {
    #Set a type so we can identify this pipeline
    type => "keyboard_input"
    #add a field where we can measure against.
    add_field => [ "fieldname", "value" ]
  }
}

filter {
  if [type] == "keyboard_input" {
    metrics {
      meter => "fieldname"
      add_tag => "metric"
    }
  }
}

output {
  # only emit events with the 'metric' tag
  if "metric" in [tags] {
    stdout {
      codec => line {
        format => "rate: %{[fieldname][rate_1m]} count: %{[fieldname][count]}"
      }
    }
  }
}

I included this for future reference.