Marvel refusing to load : "Cannot read property 'cluster_name' of undefined"

Elastic Stack Elasticsearch
1

Hi there,

I just installed Marvel for the first time on a production cluster running Kibana 4.4.2, ES 2.2.1. Followed the installation instructions, but when I try to load Marvel, I get:

Marvel: Error 400 Bad Request: Cannot read property 'cluster_name' of undefined

Calling /api/marvel/v1/clusters returns the same:

{
  statusCode: 400,
  error: "Bad Request",
  message: "Cannot read property 'cluster_name' of undefined"
}

cluster.name is defined in elasticsearch.yml and shows (as cluster_name) with a curl localhost:9200.

So I'm not quite sure what's going on. The Marvel agent is definitely running, and I have the Marvel indices created (I already tried deleting them and recreating them):

.marvel-es-2016.03.16
.marvel-es-2016.03.17
.marvel-es-data

I also tried stopping all the nodes and restarting them. Ideas?

Thanks!

2

Hi Julien,

Can you show the output of

curl localhost:9200/_cat/indices/.marvel*?v

I'm wondering if your .marvel-es-data index is not actually being populated. Can you verify that you master node(s) have all been restarted after installing Marvel? You can verify this by running:

curl localhost:9200/_cat/plugins?v

Any node not listing Marvel has not been restarted (or it wasn't installed).

3

Hey Chris,

Here is the output :

$ curl localhost:9200/_cat/indices/.marvel*?v
health status index                 pri rep docs.count docs.deleted store.size pri.store.size
green  open   .marvel-es-data         1   1         10         1470     28.6kb         14.3kb
green  open   .marvel-es-2016.03.21   1   1    1835434         3908    594.9mb        290.7mb

And :

$ curl localhost:9200/_cat/plugins?v
name           component    version type url
IP1            license      2.2.1   j
IP1            marvel-agent 2.2.1   j
IP2            license      2.2.1   j
IP2            marvel-agent 2.2.1   j
IP3            license      2.2.1   j
IP3            marvel-agent 2.2.1   j

Everything looks ok, though I’m not sure what the doc count is supposed to be in .marvel-es-data.

4

Alright, so you've got some data and your nodes all appear to have it running.

Can you fetch the results from the .marvel-es-data index so that we can see what's going on in there?

curl -XGET localhost:9200/.marvel-es-data/cluster_info/_search?pretty -o cluster_info.json

I assume that this will be kind of large or it will be empty. One or the other. Assuming it's large, can you copy/paste the output of it into a GitHub gist, then link to it. If it's empty, then just let me know that too.

Thanks

5

Hey Chris,

It was mostly empty:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : ".marvel-es-data",
      "_type" : "cluster_info",
      "_id" : "qbRu5dJtRGKQcurrMduJlQ",
      "_score" : 1.0
    } ]
  }
}

Though looking at elasticsearch-head, in the browser section, I can see that there are 9 "node" documents, all fairly similar in terms of fields:

{
  "_index": ".marvel-es-data",
  "_type": "node",
  "_id": "tai6pwhESQaofOuLLe9Hmg",
  "_version": 328,
  "_score": 1
}

Thanks for helping me debug this by the way, I really appreciate it :slight_smile:

6

Hmm. This definitely looks wrong. Even the _id of that document looks wrong to be honest.

For reference, here's my own with the license part removed. You will probably immediately see where cluster_name comes from and hence the error you're seeing.

Do you have any global templates in play? Can you list the response for

curl -XGET localhost:9200/_template?pretty -o templates.json

Thanks.

7

Ha... Good catch! I do have a global template. This cluster stores timeseries, so the default template is optimized for space and performance (no source in particular).

Unfortunately, the index names don't have a clear pattern I can use, except that they don't start with .. Is there a simple pattern I can use to include all indices except the .xxx ones?

8

Hi Julien,

Unfortunately not. At least not yet! There's no "exclusionary" part of templates, so I created an issue for it. However, this is definitely the issue. Would it be possible to list that template manually for your indices? I realize that may be a nuisance, but it will allow Marvel indices (and Kibana ones for that matter) to work.

Hopefully we can add a way to hide indices from the templates to avoid this problem.

As an aside about your global template:

        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "doc_values" : true,
              "type" : "string"
            },
            "match_mapping_type" : "string",
            "match" : "*"
          }
        } ],
        "properties" : {
          "@timestamp" : {
            "doc_values" : true,
            "type" : "date"
          }
        },

As-of ES 2.0, doc_values defaults to true for everything except analyzed strings, so you can drop the "doc_values" : true from your mappings.

9

Hey Chris,

Thanks for all the help! I'll figure something out for the templates. I'm just happy to have found the root cause :slight_smile:

I'll update again once I have everything working.

Cheers!

1 Like
10

Alright, it took me a minute, but I finally got everything working : deleted my wildcard template and used specific templates instead. Marvel now works as expected.

Thanks again.

Cheers!