Marvel only shows data from master node

Elastic Stack Elasticsearch
1

Hi there,

we did an updgrade from ES 2.2.1 -> ES 2.3.1, Kibana 4.4.2 -> Kibana 4.5.0 and using the latest Marvel version. Agent is installed on each node (incl. license) and after start of Kibana we only get data from the master node. Same after switching master to different node:

marvel_issue.png1869×275 19.6 KB

Can anyone help me, what I have to configure to see data for each node again?

Regards,
Marcell

2

Hi Marcell,

This looks to be the same issue as in All nodes except Master show as "Offline" .

This is the result of the new node resolver feature not picking up the transport address properly. You can workaround this issue by changing the node resolver to use the node's name as the unique properly rather than the transport address (the default). This is done in the Kibana config (kibana.yml):

marvel.node_resolver: name

Given that you do not have have domain names in your transport address, can I have you run this to ensure that the forthcoming fix is enough:

GET /.marvel-es-1*/node_stats/_search
{
   "size" : 0,
   "aggs" : {
      "nodes" : {
         "date_histogram" : {
            "interval" : "10s",
            "field" : "timestamp",
            "order" : {
               "_key" : "desc"
            },
            "min_doc_count" : 1
         },
         "aggs" : {
            "source_node_name" : {
               "terms" : {
                  "field" : "source_node.name"
               },
               "aggs" : {
                  "source_node_transport_address" : {
                     "terms" : {
                        "field" : "source_node.transport_address"
                     }
                  }
               }
            }
         }
      }
   }
}
3

Hi Chris,

thanks. The workaround fixed the issue. The search request succeeded and returned plenty of lines:

"aggregations": {
    "nodes": {
      "buckets": [
        {
          "key_as_string": "2016-04-14T15:51:50.000Z",
          "key": 1460649110000,
          "doc_count": 1,
          "source_node_name": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "elastic04",
                "doc_count": 1,
                "source_node_transport_address": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                    {
                      "key": "elastic04/10.30.0.118:9300",
                      "doc_count": 1
                    }
                  ]
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2016-04-14T15:51:40.000Z",
          "key": 1460649100000,
          "doc_count": 3,
          "source_node_name": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "elastic04",
                "doc_count": 1,
                "source_node_transport_address": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                    {
                      "key": "elastic04/10.30.0.118:9300",
                      "doc_count": 1
                    }
                  ]
                }
              },
              {
                "key": "elastic05",
                "doc_count": 1,
                "source_node_transport_address": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                    {
                      "key": "elastic05/10.30.0.119:9300",
                      "doc_count": 1
                    }
                  ]
                }
              },
              {
                "key": "elastic06",
                "doc_count": 1,
                "source_node_transport_address": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                    {
                      "key": "elastic06/10.30.0.120:9300",
                      "doc_count": 1
                    }
                  ]
                }
              }
            ]
          }
        },
...

Hope, the excerpt of the response can help you verifying.

Regards,
Marcell

4

Thanks Marcell,

It looks like your issue's cause is the same as the others, which is a great sign.

5

Thanks Chris.

PS: Btw, what tags do I have to use in order to format the json I posted ? :wink:

6

Single ` are meant for a single line. You needed to use ``` (on their own lines). Like this:

```
block
of
code
```

Rather than

`block
of
code`