Monitoring -> Elasticsearch -> Nodes - no data nodes(

Elastic Stack Kibana
1

Hello World!

... while trying to navigate through Monitoring -> Elasticsearch -> Nodes, I've noticed that some of nodes are missing from the list (even though node count on top is correct), all nodes that are missing are dedicated data nodes...

I've had same issue with 6.5.4 and now same behavior with 6.6.0.

Please advise.

2

Hi @alexus,

That list is derived from what we see in the .monitoring-es-6-* monitoring documents using the type: node_stats.

Try running this query and report back your findings:

POST .monitoring-es-6-*/_search
{
    "size": 1,
    "query": {
        "term": {
            "type": "node_stats"
        }
    },
    "sort": {
        "timestamp": {
            "order": "desc"
        }
    },
    "collapse": {
        "field": "source_node.uuid"
    },
    "aggs": {
    	"nodes": {
    		"terms": {
    			"field": "source_node.uuid"
    		}
    	}
    }
}
3

I ran query that you provided and only saw single node in output, I then tried to run it few more times, and every time I got different node (didn't see any of nodes that are missing from the list I mentioned earlier).

4

What is the Elasticsearch monitoring configuration for these nodes? Are you using a dedicated monitoring cluster? Are the nodes exporting to the right monitoring cluster? Is Kibana pointed to the right monitoring cluster too?

See:

5

in regards to monitoring, all nodes are configured in same way, yet some nodes are not showing up in Monitoring.

ATM, monitoring is done in same cluster as nodes reside, same goes for Kibana (part of same cluster)

6

Can you show me the cluster settings for each node and then let me know which node is the elected master?

7

Per Cluster Get Settings | Elasticsearch Reference [6.6] | Elastic

{
  "persistent" : { },
  "transient" : { }
}

there are few of each nodes: master elected, data, ingest and coordinate only nodes, they all part of same elasticsearch cluster, pretty standard... there is no custom configuration really...

8

Hi @alexus, modifying the query Chris provided above a bit, could you run it and post the results please?

POST .monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id
{
  "query": {
    "term": {
      "type": "node_stats"
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

Thanks!

9

I copied @shaunak query into $$.json file and ran query (w/ json file) against elasticsearch cluster few times:

What is odd here is that every time I query cluster, I get different answer from last run:

# cat $$.json | curl --silent --header 'Content-Type: application/json' --request POST "$ELASTICSEARCH_URI/.monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id&pretty" --data @- | grep esm
            "name" : "esm5",
            "name" : "esm1",
# cat $$.json | curl --silent --header 'Content-Type: application/json' --request POST "$ELASTICSEARCH_URI/.monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id&pretty" --data @- | grep esm
            "name" : "esm4",
            "name" : "esm5",
#

esmX is hostname of elasticsearch node, where X is number of the node. I ran it few times, however data node never appeared in the list...

10

Okay, changing the query slightly again:

POST .monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id,hits.hits._source.cluster_uuid
{
  "query": {
    "term": {
      "type": "node_stats"
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

Also, this time would you mind posting the entire results of your query, without passing them through grep? Of course, feel free to mask any publicly addressable IP addresses.

11

BTW, you don't have to issue curl commands for these queries. You can open Kibana > Dev Tools
(click on the wrench/spanner icon in the left nav) > Console and paste my queries as-is.

12

if you don't mind, could you please tell me what exactly are you hoping to find in my output? there is no publicly addressable IPs as I use private range, most sensetive information there is name, uuid and node_id?

13

The grep'd output tells me that that the names are changing each time, which is definitely odd. So I was hoping to see if other properties of the same document are also changing as well.

14

full output: #1 & #2 runs...

following is example of same node that is present in both runs - no changes

"_source" : {
  "cluster_uuid" : "01ZhYrrJRuyXlvTAIVTKpg",
  "source_node" : {
    "uuid" : "33Rmo_UURXSNhEWb3KUzLw",
    "host" : "10.0.2.49",
    "transport_address" : "10.0.2.49:9300",
    "ip" : "10.0.2.49",
    "name" : "esc3",
    "timestamp" : "2019-02-06T21:17:32.430Z"
  },
  "node_stats" : {
    "node_id" : "33Rmo_UURXSNhEWb3KUzLw"
  }
}
15

Results from both runs showed exactly 10 hits. How many total nodes (including the data nodes that are not showing up) do you have in your cluster? Any chance you could provide the result of GET _cat/nodes?v next? Thanks!

16

total count of nodes over 20, each output includes 10 random nodes..

esc5 is part of 2nd run, yet not included in 1st run while esm5 is part of 1st run, however not part of 2nd and neither of run has esdX - which are data nodes.

as I had mentioned earlier, all nodes appeared fine prior to 6.5.4 version, not sure if issue is on my end to be honest...

17

This would be really helpful for me to get a clear picture on the nodes in your cluster, if you can provide it.

18

https://pastebin.com/BvcLh1eQ

19

Thanks for sharing that. I looked through the _cat/nodes output and compared it with the results of the two runs you posted earlier. In addition to the 5 data nodes never appearing the query results, I found a couple other nodes not appearing either.

Perhaps we are running into a default size limit of the collapsed ES search query, so lets try a tweaked query with an explicit large enough size:

POST .monitoring-es-*/_search?filter_path=hits.hits._source.source_node,hits.hits._source.node_stats.node_id,hits.hits._source.cluster_uuid
{
  "size": 30,
  "query": {
    "term": {
      "type": "node_stats"
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

As before, please post the entire results (pastebin is great). Thanks!

20

https://pastebin.com/DXFaSZG3