# One ES node not showing in Kibana Monitoring

**URL:** <https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212>\
**Category:** Kibana\
**Tags:** elastic-stack-monitoring\
**Created:** [September 19, 2019, 1:47pm UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212 "2019-09-19T13:47:21Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![thealy](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@thealy](https://discuss.elastic.co/u/thealy)\
**Post date:** [November 5, 2019, 2:06pm UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/21 "2019-11-05T14:06:29Z")

</div>

My problem turned out to be mis-match of time setting on the system that was not showing. Try running ntpupdate or install ntp if not already.

---

<div class="post-metadata">

**Author:** ![sharry007](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sharry007](https://discuss.elastic.co/u/sharry007)\
**Post date:** [November 6, 2019, 5:59am UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/22 "2019-11-06T05:59:08Z")

</div>

As per the output of the first query, node id for upgraded master is "m\_7bmIwbRoi-j80tDms2lw" and node id for upgraded warm node is "a5\_1IeuxSji6QYlo3SHQ6Q".

Output of query `GET /_nodes?filter_path=nodes.*.name` is

```
{
  "nodes" : {
    "m_7bmIwbRoi-j80tDms2lw" : {
      "name" : "master-1"
    },
    "A8Qe1_q3QUqYPsgw3jcz5g" : {
      "name" : "hot-2"
    },
    "gdoeR2EzTvCuE4RDeeb_aA" : {
      "name" : "hot-3"
    },
    "XZFg8VpuRS2WIse9KBIvMA" : {
      "name" : "coordinate-1"
    },
    "yvOSip5mQRG3uThwezXGBg" : {
      "name" : "WARM-2"
    },
    "JNSXpU_1SP-nYn5lxWxZXQ" : {
      "name" : "hot-4"
    },
    "ebvjKCrhQvK8581Uvr8Gew" : {
      "name" : "master-2"
    },
    "GdXTW8qmR2aDXclXrQAgRw" : {
      "name" : "coordinate-3"
    },
    "Qfpu2zrJTHaP_ngUNLZ7Jw" : {
      "name" : "hot-1"
    },
    "xDtX5E1JRnu2BNv8zHe8bQ" : {
      "name" : "coordinate-2"
    },
    "djCtwvYuSgWcgQ2DnbvitQ" : {
      "name" : "master-3"
    },
    "2OazDzlySLWXeTlNlkQ-cA" : {
      "name" : "WARM-2"
    },
    "fz14QsFFRYijYJqOWRVONw" : {
      "name" : "warm-1"
    },
    "XYIU3i3cRw-C7jAdDRj17g" : {
      "name" : "WARM-4"
    }
  }
}

```

The upgraded nodes are master-1 and warm-1. Output of above query and `GET /_nodes/stats` show node id for warm-1 to be "fz14QsFFRYijYJqOWRVONw" whereas output of

```
    GET .monitoring-es-*/_search?filter_path=aggregations.node_ids
    {
    "size": 0,
    "query": {
    "term": {
    "type": {
    "value": "node_stats"
    }
    }
    },
    "aggs": {
    "node_ids": {
    "terms": {
    "field": "node_stats.node_id",
    "size": 50
    }
    }
    }
    }

```

doesn't contain "fz14QsFFRYijYJqOWRVONw". Instead it contains "a5\_1IeuxSji6QYlo3SHQ6Q" as one of the node ids.

---

<div class="post-metadata">

**Author:** ![sharry007](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sharry007](https://discuss.elastic.co/u/sharry007)\
**Post date:** [November 6, 2019, 8:46am UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/23 "2019-11-06T08:46:22Z")

</div>

I found following logs in the upgraded master node:

`[o.e.x.m.e.l.LocalExporter] [master-1] monitoring index template [.monitoring-logstash] does not exist, so service cannot start (waiting on master)`

`[o.e.x.m.e.l.LocalExporter] [master-1] waiting for elected master node [{master-3}{djCtwvYuSgWcgQ2DnbvitQ}{sdvoobvCS6ig6brEOUmMPA}{<ip>}{<ip>:9300}{xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)`

`[o.e.x.m.e.Exporters] [master-1] skipping exporter [default_local] as it is not ready yet`

The last one keeps repeating every 10 seconds.

X-pack monitoring is enabled on all the nodes.

I saw below issues and it seems like I'm facing problems due to different versions of elasticsearch.  
[skipping exporter [default\_local] as it is not ready yet](https://discuss.elastic.co/t/info-o-e-x-m-e-exporters-es-node1-skipping-exporter-default-local-as-it-is-not-ready-yet/73735)  
[Rebuilt node, logs say waiting for local exporter](https://discuss.elastic.co/t/solved-rebuilt-node-logs-say-waiting-for-local-exporter/92482)  
[Rolling upgrade sequence of node types](https://discuss.elastic.co/t/rolling-upgrade-sequence-of-node-types/120311/6)

Will the issue be resolved if I upgrade all the nodes to 6.8.3?

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [November 6, 2019, 2:33pm UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/24 "2019-11-06T14:33:00Z")

</div>

> [@sharry007](#):
>
> {master-3}{djCtwvYuSgWcgQ2DnbvitQ}

Is there anything in the logs on this node indicating an issue? That's the elected master and expected to create the necessary prerequisites for monitoring to work properly (in this case, index templates)

It might be worth seeing what templates exist and which don't, but running:

```auto
GET _template/.monitoring*

```

---

<div class="post-metadata">

**Author:** ![sharry007](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sharry007](https://discuss.elastic.co/u/sharry007)\
**Post date:** [November 7, 2019, 8:32am UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/25 "2019-11-07T08:32:25Z")

</div>

> [@chrisronline](#):
>
> Is there anything in the logs on this node indicating an issue? That's the elected master and expected to create the necessary prerequisites for monitoring to work properly (in this case, index templates)

I found following in the elected master

`[o.e.c.s.ClusterApplierService] [master-3] processing [apply cluster state (from master [master {master-3}{djCtwvYuSgWcgQ2DnbvitQ}{sdvoobvCS6ig6brEOUmMPA}{<ip>}{<ip>:9300}{xpack.installed=true} committed version [91269] source [zen-disco-node-join[{master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true}]]])]: took [25ms] done applying updated cluster state (version: 91269, uuid: gsZ5opT3R3uTcaJ_mpVChA)`  
`[o.e.c.s.MasterService] [master-3] processing [zen-disco-node-join[{master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true}]]: took [4.8s] done publishing updated cluster state (version: 91269, uuid: gsZ5opT3R3uTcaJ_mpVChA)`  
`[o.e.c.s.MasterService] [master-3] processing [master ping (from: {master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true})]: execute`  
`[o.e.c.s.MasterService] [master-3] processing [master ping (from: {master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true})]: took [0s] no change in cluster state`  
`[o.e.x.m.e.l.LocalExporter] [master-3] monitoring index templates and pipelines are installed on master node, service can start`

I did not find any error in the logs except the following:

`[c.f.s.r.IndexResolverReplacer] [master-3] class org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest not supported (It is likely not a indices related request)`

---

<div class="post-metadata">

**Author:** ![sharry007](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sharry007](https://discuss.elastic.co/u/sharry007)\
**Post date:** [November 7, 2019, 8:50am UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/26 "2019-11-07T08:50:31Z")

</div>

> [@sharry007](#):
>
> [o.e.x.m.e.l.LocalExporter] [master-1] monitoring index template [.monitoring-logstash] does not exist, so service cannot start (waiting on master)

Since, we found above logs in the upgraded master, I am posting the output of `GET _template/.monitoring-logstash`

> <https://gist.github.com/sharry007/099118cdb8b9c52ab773011ed8160d42>

---

<div class="post-metadata">

**Author:** ![sharry007](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sharry007](https://discuss.elastic.co/u/sharry007)\
**Post date:** [November 7, 2019, 2:24pm UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/27 "2019-11-07T14:24:22Z")

</div>

I have upgraded all the nodes to 6.8.3 and now kibana monitoring is showing all the nodes. I'll upgrade kibana to 6.8.3.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2019, 2:24pm UTC](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212/28 "2019-12-05T14:24:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/one-es-node-not-showing-in-kibana-monitoring/200212.md?page=1)
