Kibana display zero shards in Monitoring nodes tab

Hi Elastic team,

I'm using Elastic Stack 7.2.0.

Kibana displays zero shards for each node in the monitoring tab.
But it can display total shards correctly.

What is the problem?

Are they data nodes?

Yes, I have 12 data nodes. 9 for hot and 3 for warm.

Hi @worapojc,

We retrieve information about shards from a certain type of document (type=shards) in .monitoring-es-* indices.

Let's run a query to get a better sense of what that data looks like in your system. I'm assuming you're only monitoring a single cluster, and that you're seeing these issues still. If either of those are untrue, we'll need to adjust the query slightly, but otherwise run this and report back on the results:

POST .monitoring-es-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": {
              "value": "shards"
            }
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-1h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "nodes": {
      "terms": {
        "field": "shard.node",
        "size": 10000
      }
    }
  }
}

Hi @chrisronline, here is the result.

{
  "took" : 65,
  "timed_out" : false,
  "_shards" : {
    "total" : 16,
    "successful" : 16,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "nodes" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "7DxPvJW4QAm_TrmY9UOXjg",
          "doc_count" : 177235
        },
        {
          "key" : "LIwkxHZSTe61eFOcDmjAbw",
          "doc_count" : 158810
        },
        {
          "key" : "LcwOpRiSQny_ax7bgYUTUQ",
          "doc_count" : 155615
        },
        {
          "key" : "Z35hmlz_Q2GAGhmLJRAO4w",
          "doc_count" : 70706
        },
        {
          "key" : "TdgJhD7DSzS-gYN0Tr6qkA",
          "doc_count" : 68855
        },
        {
          "key" : "P68fR5riQ72V5rLyfoidBQ",
          "doc_count" : 50162
        },
        {
          "key" : "3i6_-lw3RdK0CDxSaueYXg",
          "doc_count" : 36630
        },
        {
          "key" : "4MqQ-MR9RKyT81-KzFC2hQ",
          "doc_count" : 36630
        },
        {
          "key" : "FRsLS5v4RmiN1s1WgSA5Pw",
          "doc_count" : 36630
        },
        {
          "key" : "05NFJrtBSJOUlTfaunJ-vQ",
          "doc_count" : 36580
        },
        {
          "key" : "je3jrr1cT8e4_5za6MhsfQ",
          "doc_count" : 36538
        },
        {
          "key" : "YMH2mvQ_QuSYOJz_IlAEaQ",
          "doc_count" : 36533
        },
        {
          "key" : "LcHh6zNFSNmm8PiD27yXRA",
          "doc_count" : 36445
        },
        {
          "key" : "QRs7ZpxqRwCuLiK-n2DDJw",
          "doc_count" : 36445
        },
        {
          "key" : "LDkWZ9O9T1KazwDidZudIA",
          "doc_count" : 36401
        }
      ]
    }
  }
}

I noticed that there is zero shard in the document with shards type.

     {
        "_index" : ".monitoring-es-7-2019.08.16",
        "_type" : "_doc",
        "_id" : "IMoJ84inSD-ifycLpRKATg:FRsLS5v4RmiN1s1WgSA5Pw:a202171-windows_event_log-2019.33:0:p",
        "_score" : 1.0236014,
        "_source" : {
          "cluster_uuid" : "z_eKGqwoTV6H6cOaf5__Ow",
          "timestamp" : "2019-08-16T03:49:28.397Z",
          "interval_ms" : 10000,
          "type" : "shards",
          "source_node" : {
            "uuid" : "FRsLS5v4RmiN1s1WgSA5Pw",
            "host" : "10.49.116.247",
            "transport_address" : "10.49.116.247:9300",
            "ip" : "10.49.116.247",
            "name" : "10.49.116.247",
            "timestamp" : "2019-08-16T03:49:28.246Z"
          },
          "state_uuid" : "IMoJ84inSD-ifycLpRKATg",
          "shard" : {
            "state" : "STARTED",
            "primary" : true,
            "node" : "FRsLS5v4RmiN1s1WgSA5Pw",
            "relocating_node" : null,
            "shard" : 0,
            "index" : "a202171-windows_event_log-2019.33"
          }
        }
      }

I configured the data cluster to send the monitoring by xpack monitoring.

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.monitoring.exporters:
    id1:
        type: http
        host: <list of montior hosts>

I also disabled xpack monitoring on the monitoring cluster, what's the cause of this issue?