Http_poller -> graphite

Elastic Stack Logstash
1

Trying to use the new input-http_poller. I've got an application which on a url, spits out json which looks like this in stdout/rubydebugger:

{
  "metrics" =>  {
     "requests" => 12,
     "foo" => 2345,
     "bar" => 999
  }
}

Is there an easy way to say in a graphite output block: "Take everything under 'metrics', and send to graphite?

2

Digging deeper into the graphite plugin, it looks like something like this should work:

if [http_poller_metadata] {
    graphite {
      fields_are_metrics => true
      include_metrics => [".*metrics.*"]
      metrics_format => "poller.%{host}.*"
    }
  }

But it doesn't appear to be sending anything.

3

Doing some debugging with:

stdout {
      codec => graphite {
        fields_are_metrics => true
        metrics_format => "poller.%{host}.*"
      }
    }

It appears that it doesn't deal with nested fields at all. All it displays output for is the top-level field. Nothing deeper.

4

So I think I can make it work if I do two things:

  1. Just drop all fields except for the nested metric field.
  2. Iterate over the metric field, and create new fields/values based on them.

Does anyone have ruby code that can do this in a filter, if I assume the 'metric' field is never more than 1 level deep?

5

Hey Bruce, I'll take a look at this later today!

6

Thanks!

I think this sort of thing could make http-poller super useful, in conjunction with graphite.

7

I also created this enhancement request for the graphite output plugin:

8

Thanks Bruce. I agree that the root cause is missing functionality in the graphite output. I'll try and put a patch in today.

9

This has been fixed in the 1.0.1 release of the graphite output. Thanks bringing up the issue guys!

10

Thanks for the quick turn around! I'm trying to use it right now.

11

Not working yet. Here's what I see in logstash.stdout from http-poller. (Trimming out other stuff.)

{
                "MetricsC" => {
        "requests" => 3
    },
}

Here's my config:

input {
  http_poller {

    urls => {
      test => "https://test.com/status"
    }
    request_timeout => 30
    interval => 300
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}
filter {
  if [http_poller_metadata] {
      mutate {
        add_field => {
          "host" => "%{http_poller_metadata[name]}"
        }
      }
    }
}
output {
  if [http_poller_metadata] {
    stdout {
      codec => graphite {
        fields_are_metrics => true
        include_metrics => ["MetricsC"]
        metrics_format => "test.infra.poller.%{host}.*"
      }
    }
    graphite {
      host => "xxxxxx"
      fields_are_metrics => true
      include_metrics => ["MetricsC"]
      metrics_format => "test.infra.poller.%{host}.*"
    }
  }
  stdout {
    codec => rubydebug
  }
}

Am I doing anything wrong?

12

I think I got it to work. Didn't seem to like my %{host} in metrics format. Testing more.

13

Okay, If I do include_metrics => ["MetricsC"], that works properly.

However, I want to also get http_poller_metadata.runtime_seconds and http_poller_metadata.code.

So I tried this:

      include_metrics => ["MetricsC","http_poller_metadata"]

That works for getting everything.

How would I restrict that to just code and runtime_seconds?

14

Also, it appears that variables aren't interpolated inside of 'metrics_format':

      metrics_format => "team.infra.poller.%{host}.*"

That results in the directory %{host} in graphite.

15
16

For people following along, Andrew added that enhancement, and now metrics_format works as expected.