# All nodes except Master show as "Offline"
**URL:** https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274
**Category:** Elasticsearch
**Tags:** elastic-stack-monitoring
**Created:** [April 4, 2016, 5:45pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274 "2016-04-04T17:45:37Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 4, 2016, 5:45pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/1 "2016-04-04T17:45:37Z")
## Running
- Elasticsearch 2.3.0
- Kibana 4.5.0
- Marvel-agent 2.3.0
## Description
The nodes page in Marvel shows all 4 nodes in my cluster, with proper metadata about each (hostname, ip address, name). However all except the master node show as "offline". The master has appropriate data about disk usage, CPU, etc, but the other nodes show as "offline".
## Extra Notes
- Upgrading to marvel-agent 2.3.0 from 2.2.1 caused marvel to start storing data in the index pattern marvel-es-1-\*, whereas previously data was stored in marvel-es-\*.
- Nodes that are offline for longer than the time bounds in Marvel fall off the screen, so some metadata is still being reported by the nodes that allows them to continue to be displayed.
- Clicking on the name of any of the nodes (including the master) causes a redirection to the root of marvel, instead of a node detail page.
- No exceptions in the ElasticSearch or Kibana logs.
## Screenshot
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 4, 2016, 6:22pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/2 "2016-04-04T18:22:19Z")
Hi Chris (great name),
Can you show the output of
```auto
curl -XGET host:9200/_cat/plugins?v
```
from any of your nodes?
Thanks,
Chris
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 4, 2016, 6:34pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/3 "2016-04-04T18:34:27Z")
> [@pickypg](#):
>
> /\_cat/plugins?v
Thanks Chris :D,
```auto
curl -XGET "http://elk1.cc.pdrop.net:9200/_cat/plugins?v=1"
name component version type url
Apophis license 2.3.0 j
Apophis marvel-agent 2.3.0 j
Anubis license 2.3.0 j
Anubis marvel-agent 2.3.0 j
Hathor license 2.3.0 j
Hathor marvel-agent 2.3.0 j
Nirrti license 2.3.0 j
Nirrti marvel-agent 2.3.0 j
```
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 4, 2016, 7:23pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/4 "2016-04-04T19:23:55Z")
Just to be sure, can you verify that you have the proper version of the Kibana plugin?
```auto
$ grep version installedPlugins/marvel/package.json
```
If that all matches up, then the next step is to crack open the index to start to see what's there and not there.
Thanks,
Chris
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 4, 2016, 7:40pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/5 "2016-04-04T19:40:37Z")
> [@pickypg](#):
>
> version installedPlugins/marvel/package.json
Hey Chris,
Thanks for your help:
```auto
root@500514ad55df:/opt/kibana# grep version installedPlugins/marvel/package.json
"version": "2.3.0",
```
---
### Author:  [@Jakauppila](https://discuss.elastic.co/u/Jakauppila)
#### Post date: [April 5, 2016, 8:43pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/6 "2016-04-05T20:43:01Z")
I'm seeing the same thing as Chris, except that I can click on the names of my nodes and get the current stats on them (although I can't see the shard allocations on the nodes that are not master).
I'm running shield, but I'm using the local exporter on Marvel.
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 5, 2016, 9:24pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/7 "2016-04-05T21:24:24Z")
I'm not running Shield, just all the free stuff. :\
---
### Author:  [@Jakauppila](https://discuss.elastic.co/u/Jakauppila)
#### Post date: [April 5, 2016, 9:47pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/8 "2016-04-05T21:47:05Z")
I turned on DEBUG for Marvel, but I'm not seeing any errors on any of the nodes, just the following:
```auto
[2016-04-05 16:27:32,187][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 0, value: .marvel-es-1]] in version [1]
[2016-04-05 16:27:32,187][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 4, value: .marvel-es-data-1]] in version [1]
[2016-04-05 16:27:54,391][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 0, value: .marvel-es-1]] in version [1]
[2016-04-05 16:27:54,391][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 4, value: .marvel-es-data-1]] in version [1]
[2016-04-05 16:27:55,594][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 0, value: .marvel-es-1]] in version [1]
[2016-04-05 16:27:55,594][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 4, value: .marvel-es-data-1]] in version [1]
[2016-04-05 16:27:55,797][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 0, value: .marvel-es-1]] in version [1]
[2016-04-05 16:27:55,797][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 4, value: .marvel-es-data-1]] in version [1]
[2016-04-05 16:27:56,000][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 0, value: .marvel-es-1]] in version [1]
[2016-04-05 16:27:56,000][DEBUG][marvel.agent.exporter.local] found index template [[cursor, index: 4, value: .marvel-es-data-1]] in version [1]
```
Looking within the `.marvel-es-1-*` index appears that I'm getting the data from each of the nodes, but that it's just not displaying them on the node overview page.
---
### Author:  [@Grazius](https://discuss.elastic.co/u/Grazius)
#### Post date: [April 6, 2016, 2:52pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/9 "2016-04-06T14:52:46Z")
Hi, Same problem for me after Migration from 2.2 to 2.3.1
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 6, 2016, 3:07pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/10 "2016-04-06T15:07:52Z")
@tebriel: Since you were the first to report it, let's continue with your data as it sounds like it may be a wider issue.
Can you run this query against your cluster and attach the response?
```auto
GET /.marvel-es-1*/node_stats/_search
{
"size" : 0,
"aggs" : {
"nodes" : {
"date_histogram" : {
"interval" : "10s",
"field" : "timestamp",
"order" : {
"_key" : "desc"
},
"min_doc_count" : 1
},
"aggs" : {
"source_node_name" : {
"terms" : {
"field" : "source_node.name"
},
"aggs" : {
"source_node_transport_address" : {
"terms" : {
"field" : "source_node.transport_address"
}
}
}
}
}
}
}
}
```
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 6, 2016, 4:09pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/11 "2016-04-06T16:09:15Z")
> [@pickypg](#):
>
> /.marvel-es-1\*/node\_stats/\_search?\_source=cluster\_state.nodes
Thanks!
```auto
{
"_shards": {
"failed": 0,
"successful": 6,
"total": 6
},
"hits": {
"hits": [
{
"_id": "AVPS6Hs_HfEfRmF9lCyp",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPS6A-6HfEfRmF9lCoZ",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlVhXir2ZpyJFW7VZ",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlgxcir2ZpyJFW7Vc",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlnQ6ir2ZpyJFW7Ve",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlOzsir2ZpyJFW7VX",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlSQAir2ZpyJFW7VY",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlY0yir2ZpyJFW7Va",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlc2yir2ZpyJFW7Vb",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
},
{
"_id": "AVPSlj3rir2ZpyJFW7Vd",
"_index": ".marvel-es-1-2016.04.01",
"_score": 1.0,
"_source": {},
"_type": "node_stats"
}
],
"max_score": 1.0,
"total": 122720
},
"timed_out": false,
"took": 696
}
```
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 6, 2016, 4:23pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/12 "2016-04-06T16:23:49Z")
@tebriel: Can you try resending the request as a `POST`? It looks like whatever tool you used stripped out the request's body (web browsers do not natively support sending `GET` requests with a body).
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 6, 2016, 6:27pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/13 "2016-04-06T18:27:58Z")
@Jakauppila: To try a different approach, you are clearly naming your nodes. Can you try adding this setting to your _Kibana_ configuration, then restart Kibana, to see if it resolves the issue?
```auto
marvel.node_resolver: name
```
For anyone interested, [the `marvel.node_resolver` setting was added in Marvel 2.3](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/11). The default -- and only other -- value for it is `transport_address`.
I have had no luck reproducing this locally, but I want to see if this at least resolves it, which will point to a problem.
---
### Author:  [@Jakauppila](https://discuss.elastic.co/u/Jakauppila)
#### Post date: [April 6, 2016, 6:34pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/14 "2016-04-06T18:34:01Z")
That looks to have fixed my problem! I can see the node info on the overview page as well as seeing the index/shard info when clicking into each node.
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 6, 2016, 9:30pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/15 "2016-04-06T21:30:25Z")
So, I reran the query as a POST (duh, sorry, was in a meeting and didn't think about that my rest client won't send a body on a GET). [The response](https://gist.github.com/tebriel/22d5a44700c1e120dbf6ccfe7b24c1c3) is huge.
I'll try using the name\_resolver next.
Thanks
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 6, 2016, 9:32pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/16 "2016-04-06T21:32:22Z")
Hi @tebriel,
Given that your node names have unique, static names (I had to check that they weren't in the default list because Greek god names are so close sometimes!), I'm hopeful that it also fixes it for you. I'm looking over the data though.
Thanks!
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 6, 2016, 9:34pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/17 "2016-04-06T21:34:34Z")
> [@pickypg](#):
>
> marvel.node\_resolver: name
To help debug, this is how I launch Elasticsearch inside a docker:
```auto
elasticsearch -Des.cluster.name="pindrop_elk" -Des.discovery.zen.ping.unicast.hosts="elk1.cc.pdrop.net, elk2.cc.pdrop.net, elk3.cc.pdrop.net, elk4.cc.pdrop.net" -Des.node.name="Apophis" -Des.network.bind_host="0.0.0.0" -Des.network.publish_host=elk1.cc.pdrop.net -Des.node.master=true -Des.node.data=true -Des.path.data=/usr/share/elasticsearch/data
```
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 6, 2016, 9:44pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/18 "2016-04-06T21:44:30Z")
Yes, using the `name_resolver` does fix the issue for now, thanks!
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 6, 2016, 9:45pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/19 "2016-04-06T21:45:59Z")
Awesome. As long as your transport address isn't changing (I noticed that it change for Jared), then it shouldn't be required for you to use this setting, but I'm glad that it resolves the issue.
We're digging into the root cause and we'll hopefully a fix out in the next release.
---
### Author:  [@tebriel](https://discuss.elastic.co/u/tebriel)
#### Post date: [April 6, 2016, 9:46pm UTC](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274/20 "2016-04-06T21:46:39Z")
Sweet, thanks! I'm pushing the updated kibana config out. Appreciate your help.
[Next page](https://discuss.elastic.co/t/all-nodes-except-master-show-as-offline/46274.md?page=2)