# Monitoring -\> Elasticsearch -\> Nodes - no data nodes(

**URL:** <https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321>\
**Category:** Kibana\
**Tags:** elastic-stack-monitoring\
**Created:** [February 6, 2019, 3:50pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321 "2019-02-06T15:50:08Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 3:50pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/1 "2019-02-06T15:50:09Z")

</div>

Hello World!

... while trying to navigate through `Monitoring` -\> `Elasticsearch` -\> `Nodes`, I've noticed that some of nodes are missing from the list (even though node count on top is correct), all nodes that are missing are dedicated data nodes...

I've had same issue with 6.5.4 and now same behavior with 6.6.0.

Please advise.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [February 6, 2019, 4:15pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/2 "2019-02-06T16:15:08Z")

</div>

Hi @alexus,

That list is derived from what we see in the `.monitoring-es-6-*` monitoring documents using the `type: node_stats`.

Try running this query and report back your findings:

```auto
POST .monitoring-es-6-*/_search
{
    "size": 1,
    "query": {
        "term": {
            "type": "node_stats"
        }
    },
    "sort": {
        "timestamp": {
            "order": "desc"
        }
    },
    "collapse": {
        "field": "source_node.uuid"
    },
    "aggs": {
    	"nodes": {
    		"terms": {
    			"field": "source_node.uuid"
    		}
    	}
    }
}

```

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 4:48pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/3 "2019-02-06T16:48:48Z")

</div>

I ran query that you provided and only saw single node in output, I then tried to run it few more times, and every time I got different node (didn't see any of nodes that are missing from the list I mentioned earlier).

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [February 6, 2019, 5:48pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/4 "2019-02-06T17:48:21Z")

</div>

What is the Elasticsearch monitoring configuration for these nodes? Are you using a dedicated monitoring cluster? Are the nodes exporting to the right monitoring cluster? Is Kibana pointed to the right monitoring cluster too?

See:

- [https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-settings.html)
- [https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html](https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html)

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 6:01pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/5 "2019-02-06T18:01:31Z")

</div>

in regards to monitoring, all nodes are configured in same way, yet some nodes are not showing up in `Monitoring`.

ATM, monitoring is done in same cluster as nodes reside, same goes for Kibana (part of same cluster)

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [February 6, 2019, 6:14pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/6 "2019-02-06T18:14:49Z")

</div>

Can you show me the cluster settings for each node and then let me know which node is the elected master?

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 8:24pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/7 "2019-02-06T20:24:34Z")

</div>

Per [Cluster Get Settings | Elasticsearch Reference [6.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-get-settings.html)

```
{
  "persistent" : { },
  "transient" : { }
}

```

there are few of each nodes: master elected, data, ingest and coordinate only nodes, they all part of same elasticsearch cluster, pretty standard... there is no custom configuration really...

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 8:45pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/8 "2019-02-06T20:45:09Z")

</div>

Hi @alexus, modifying the query Chris provided above a bit, could you run it and post the results please?

```auto
POST .monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id
{
  "query": {
    "term": {
      "type": "node_stats"
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

```

Thanks!

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 9:00pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/9 "2019-02-06T21:00:16Z")

</div>

I copied @shaunak query into `$$.json` file and ran query (w/ json file) against elasticsearch cluster few times:

What is odd here is that every time I query cluster, I get different answer from last run:

```
# cat $$.json | curl --silent --header 'Content-Type: application/json' --request POST "$ELASTICSEARCH_URI/.monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id&pretty" --data @- | grep esm
            "name" : "esm5",
            "name" : "esm1",
# cat $$.json | curl --silent --header 'Content-Type: application/json' --request POST "$ELASTICSEARCH_URI/.monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id&pretty" --data @- | grep esm
            "name" : "esm4",
            "name" : "esm5",
# 

```

`esmX` is hostname of elasticsearch node, where X is number of the node. I ran it few times, however data node never appeared in the list...

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 9:06pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/10 "2019-02-06T21:06:01Z")

</div>

Okay, changing the query slightly again:

```auto
POST .monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id,hits.hits._source.cluster_uuid
{
  "query": {
    "term": {
      "type": "node_stats"
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

```

Also, this time would you mind posting the entire results of your query, without passing them through `grep`? Of course, feel free to mask any **publicly addressable** IP addresses.

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 9:08pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/11 "2019-02-06T21:08:50Z")

</div>

BTW, you don't have to issue `curl` commands for these queries. You can open Kibana \> Dev Tools  
(click on the wrench/spanner icon in the left nav) \> Console and paste my queries as-is.

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 9:13pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/12 "2019-02-06T21:13:27Z")

</div>

if you don't mind, could you please tell me what exactly are you hoping to find in my output? there is no publicly addressable IPs as I use private range, most sensetive information there is name, uuid and node\_id?

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 9:15pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/13 "2019-02-06T21:15:21Z")

</div>

The grep'd output tells me that that the names are changing each time, which is definitely odd. So I was hoping to see if other properties of the same document are also changing as well.

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 9:19pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/14 "2019-02-06T21:19:34Z")

</div>

full output: [#1](https://pastebin.com/1s2CE5fs) & [#2](https://pastebin.com/qJ52DBvz) runs...

following is example of same node that is present in both runs - no changes

```
"_source" : {
  "cluster_uuid" : "01ZhYrrJRuyXlvTAIVTKpg",
  "source_node" : {
    "uuid" : "33Rmo_UURXSNhEWb3KUzLw",
    "host" : "10.0.2.49",
    "transport_address" : "10.0.2.49:9300",
    "ip" : "10.0.2.49",
    "name" : "esc3",
    "timestamp" : "2019-02-06T21:17:32.430Z"
  },
  "node_stats" : {
    "node_id" : "33Rmo_UURXSNhEWb3KUzLw"
  }
}
```

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 9:27pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/15 "2019-02-06T21:27:03Z")

</div>

Results from both runs showed exactly 10 hits. How many total nodes (including the data nodes that are not showing up) do you have in your cluster? Any chance you could provide the result of `GET _cat/nodes?v` next? Thanks!

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 9:35pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/16 "2019-02-06T21:35:28Z")

</div>

total count of nodes over 20, each output includes 10 random nodes..

`esc5` is part of 2nd run, yet not included in 1st run while `esm5` is part of 1st run, however not part of 2nd and neither of run has `esdX` - which are data nodes.

as I had mentioned earlier, all nodes appeared fine prior to 6.5.4 version, not sure if issue is on my end to be honest...

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 9:58pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/17 "2019-02-06T21:58:04Z")

</div>

> [@shaunak](#):
>
> Any chance you could provide the result of `GET _cat/nodes?v`

This would be really helpful for me to get a clear picture on the nodes in your cluster, if you can provide it.

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 6, 2019, 9:59pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/18 "2019-02-06T21:59:22Z")

</div>

[https://pastebin.com/BvcLh1eQ](https://pastebin.com/BvcLh1eQ)

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 6, 2019, 10:59pm UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/19 "2019-02-06T22:59:48Z")

</div>

Thanks for sharing that. I looked through the `_cat/nodes` output and compared it with the results of the two runs you posted earlier. In addition to the 5 data nodes never appearing the query results, I found a couple other nodes not appearing either.

Perhaps we are running into a default size limit of the `collapse`d ES search query, so lets try a tweaked query with an explicit large enough size:

```auto
POST .monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id,hits.hits._source.cluster_uuid
{
  "size": 30,
  "query": {
    "term": {
      "type": "node_stats"
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

```

As before, please post the entire results (pastebin is great). Thanks!

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [February 7, 2019, 4:09am UTC](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321/20 "2019-02-07T04:09:07Z")

</div>

[https://pastebin.com/DXFaSZG3](https://pastebin.com/DXFaSZG3)

[Next page](https://discuss.elastic.co/t/monitoring-elasticsearch-nodes-no-data-nodes/167321.md?page=2)