Collectd monitoring of ElasticSearch with CURL_JSON plugin failing

Elastic Stack Elasticsearch
1

In our environment, we're using Collectd 4.10.1 (I know ... old as dirt, but its a requirement of RightScale) and we are trying to leverage the curl_json plugin to pull metrics from our ElasticSearch cluster. We have the metrics coming in properly from http://localhost:9200/_cluster/stats and http://localhost:9200/_cluster/health.

What we can't get working though are any metrics from http://localhost:9200/_nodes/_local/stats. No matter how many or few metrics I define, and no matter whether I use 'nodes/*/...' or 'nodes//...', I cannot seem to get any data from this endpoint.

Here is our entire collectd elasticsearch.conf file, and a real copy of the raw node stats json output:
https://gist.github.com/diranged/ac7d2edb45b0eba00d42

Any ideas on what might be going on? I can't seem to get any debug logs either, and unfortunately the collectd-dbg package for 4.10.1 is not available it seems like, so we're really hamstrung here.

I have verified that ..

  • Collectd is indeed making the curl call to ElasticSearch and the response is coming back properly (via strace, and tcpdump)
  • Replacing '*' with 'HYUqTMnfSgmvd87_o3UPLw' (the node id) in the config file does not help
  • Dropping all the way down to a single metric (jvm/mem/heap_used_in_bytes) does not help

Can anyone help here?

2

What version of ES are you on?

3

We're running 1.5.2 right now.

4

Hey .. we're still stuck on this issue. I've tried everything I can think of to fix it, and we cannot figure out whats happening. We're upgraded to ElasticSearch 1.7.1 now. I've narrowed down the data being returned to this:

{
    "cluster_name": "nsp-es-staging-us1",
    "nodes": {
        "zDQZnKxxQPC1uOe7qZYHcA": {
            "timestamp": 1449605835281,
            "name": "staging-us1-es-access-uswest2-5-i-bd52dd64-nsp-es-staging-us1",
            "transport_address": "inet[/10.32.17.234:9300]",
            "host": "xxx,
            "ip": [
                "inet[/10.32.17.234:9300]",
                "NONE"
            ],
            "attributes": {
                "aws_availability_zone": "us-west-2a",
                "data": "false",
                "master": "true"
            },
            "http": {
                "current_open": 8,
                "total_opened": 114
            }
        }
    }
}

and set up a simple collectd config:

LoadPlugin curl_json
<Plugin curl_json>
  <URL "http://127.0.0.1:9200/_nodes/_local/stats/http?pretty=true">
    Instance "elasticsearch"
    <Key "nodes/zDQZnKxxQPC1uOe7qZYHcA/http/current_open">
      Type "gauge"
      Instance "http_current_open"
    </Key>
  </URL>
</Plugin>

Yet, its still not working.. anyone else run into anything like this?