Node count doesn't match

Elastic Stack Elasticsearch
1

Hello,
I manage a big Elasticsearch cluster with the following details:
Elasticsearch version 5.6.6
Dedicated Master nodes - 3
Data nodes - 56
Total nodes - 59
Data nodes details :

  • baremetal;
  • 31 GB jvm heap;
  • 4 x 1.8 TB SAS;

When I use the cluster/health API it displays 52 out of 56 data nodes, same for cat/health API, but the cat/nodes API display all my cluster nodes - 56.
I checked every node for disconnects and all seem to be connected and taking traffic.
Is there a way to check if the nodes are truly connected?

2

Welcome!

I honestly don't know why health does not report all the nodes. It's weird that the cat nodes does not display 59 nodes.
But something to note:

This is a way too old. It's not even the last stable version of the 5.x branch.
Consider upgrading to 7.10. So many things happened since your version that this problem is most likely not appearing anymore in 7.10.