Http_poller -> graphite


(Bruce Lysik) #1

Trying to use the new input-http_poller. I've got an application which on a url, spits out json which looks like this in stdout/rubydebugger:

{
  "metrics" =>  {
     "requests" => 12,
     "foo" => 2345,
     "bar" => 999
  }
}

Is there an easy way to say in a graphite output block: "Take everything under 'metrics', and send to graphite?


(Bruce Lysik) #2

Digging deeper into the graphite plugin, it looks like something like this should work:

if [http_poller_metadata] {
    graphite {
      fields_are_metrics => true
      include_metrics => [".*metrics.*"]
      metrics_format => "poller.%{host}.*"
    }
  }

But it doesn't appear to be sending anything.


(Bruce Lysik) #3

Doing some debugging with:

stdout {
      codec => graphite {
        fields_are_metrics => true
        metrics_format => "poller.%{host}.*"
      }
    }

It appears that it doesn't deal with nested fields at all. All it displays output for is the top-level field. Nothing deeper.


(Bruce Lysik) #4

So I think I can make it work if I do two things:

  1. Just drop all fields except for the nested metric field.
  2. Iterate over the metric field, and create new fields/values based on them.

Does anyone have ruby code that can do this in a filter, if I assume the 'metric' field is never more than 1 level deep?


(Andrew Cholakian) #5

Hey Bruce, I'll take a look at this later today!


(Bruce Lysik) #6

Thanks!

I think this sort of thing could make http-poller super useful, in conjunction with graphite.


(Bruce Lysik) #7

I also created this enhancement request for the graphite output plugin:


(Andrew Cholakian) #8

Thanks Bruce. I agree that the root cause is missing functionality in the graphite output. I'll try and put a patch in today.


(Andrew Cholakian) #9

This has been fixed in the 1.0.1 release of the graphite output. Thanks bringing up the issue guys!


(Bruce Lysik) #10

Thanks for the quick turn around! I'm trying to use it right now.


(Bruce Lysik) #11

Not working yet. Here's what I see in logstash.stdout from http-poller. (Trimming out other stuff.)

{
                "MetricsC" => {
        "requests" => 3
    },
}

Here's my config:

input {
  http_poller {

    urls => {
      test => "https://test.com/status"
    }
    request_timeout => 30
    interval => 300
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}
filter {
  if [http_poller_metadata] {
      mutate {
        add_field => {
          "host" => "%{http_poller_metadata[name]}"
        }
      }
    }
}
output {
  if [http_poller_metadata] {
    stdout {
      codec => graphite {
        fields_are_metrics => true
        include_metrics => ["MetricsC"]
        metrics_format => "test.infra.poller.%{host}.*"
      }
    }
    graphite {
      host => "xxxxxx"
      fields_are_metrics => true
      include_metrics => ["MetricsC"]
      metrics_format => "test.infra.poller.%{host}.*"
    }
  }
  stdout {
    codec => rubydebug
  }
}

Am I doing anything wrong?


(Bruce Lysik) #12

I think I got it to work. Didn't seem to like my %{host} in metrics format. Testing more.


(Bruce Lysik) #13

Okay, If I do include_metrics => ["MetricsC"], that works properly.

However, I want to also get http_poller_metadata.runtime_seconds and http_poller_metadata.code.

So I tried this:

      include_metrics => ["MetricsC","http_poller_metadata"]

That works for getting everything.

How would I restrict that to just code and runtime_seconds?


(Bruce Lysik) #14

Also, it appears that variables aren't interpolated inside of 'metrics_format':

      metrics_format => "team.infra.poller.%{host}.*"

That results in the directory %{host} in graphite.


(Bruce Lysik) #15

(Bruce Lysik) #16

For people following along, Andrew added that enhancement, and now metrics_format works as expected.


(system) #17