Kibana monitoring shows empty nodes

[fanqiaoqing (Qiaoqing)]

Hi Elastic,

We are using the kibana 7.2.1, we have around 200 nodes in our clusters. Recently, we found the kibana monitoring shows empty nodes

I uploaded a screenshot, you can check that. Meanwhile we had try many solutions but none is work.

This is our cluster setttings.

{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : {
    "search" : {
      "max_buckets" : "65535"
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  }
}

Could someone gave us any suggestions?

Thanks,

[bhavyarm]

Hello,

Can you please check if there are any errors in your browser console on the monitoring page?

Thanks,
Bhavya

[Igor_Zaytsev]

@fanqiaoqing

Are there any errors in the Kibana (or ES) terminal console?

I'm assuming that one of the queries is either timing-out, failing, or hitting the "max buckets" error, in that API call.

Can you please try the following queries and post the results here (be sure to replace cluster_uuid with yours )

# node shards query
POST .monitoring-es*/_search
{
  "size": 0,
  "sort": {
    "timestamp": {
      "order": "desc",
      "unmapped_type": "long"
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "term": {
                  "type": "shards"
                }
              },
              {
                "term": {
                  "metricset.name": "shards"
                }
              }
            ]
          }
        },
        {
          "term": {
            "cluster_uuid": "wuXG3QJKThmajyOWMx20hw"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-12h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "nodes": {
      "terms": {
        "field": "shard.node",
        "size": 10000
      }
    }
  }
}

and

#node unassigned shards query
POST .monitoring-es*/_search
{
  "sort": {
    "timestamp": {
      "order": "desc",
      "unmapped_type": "long"
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "term": {
                  "type": "shards"
                }
              },
              {
                "term": {
                  "metricset.name": "shards"
                }
              }
            ]
          }
        },
        {
          "term": {
            "cluster_uuid": "wuXG3QJKThmajyOWMx20hw"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-12h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "indices": {
      "terms": {
        "field": "shard.index",
        "size": 10000
      },
      "aggs": {
        "state": {
          "filter": {
            "terms": {
              "shard.state": [
                "UNASSIGNED",
                "INITIALIZING"
              ]
            }
          },
          "aggs": {
            "primary": {
              "terms": {
                "field": "shard.primary",
                "size": 2
              }
            }
          }
        }
      }
    }
  }
}

Also, can you please try higher time ranges (like 24 hours ago or 48), and see if that does anything

[fanqiaoqing (Qiaoqing)]

[quote="Igor_Zaytsev, post:3, topic:249829"]

POST .monitoring-es*/_search
{
  "size": 0,
  "sort": {
    "timestamp": {
      "order": "desc",
      "unmapped_type": "long"
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "term": {
                  "type": "shards"
                }
              },
              {
                "term": {
                  "metricset.name": "shards"
                }
              }
            ]
          }
        },
        {
          "term": {
            "cluster_uuid": "wuXG3QJKThmajyOWMx20hw"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-12h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "nodes": {
      "terms": {
        "field": "shard.node",
        "size": 10000
      }
    }
  }
}

[/quote]

Thanks Igor for youe suggestion.
I tried the two queries to get the data in past 4800, they are showed below.
1st:

2nd

The number of characters in the query result exceeds the limit, I can only give a screenshot.

and I also uploaded the screenshot of the console,

[fanqiaoqing (Qiaoqing)]

Thanks Bhavya,

I took a screenshot when loading the nodes, no obvious abnormalities found.

[Igor_Zaytsev]

@fanqiaoqing Still need a couple of things:

• Check Kibana and ES terminal console logs (not the browser) to see if there are any errors there?
• And, also try the 48h time-range in the UI time picker (top-right). Play around with it see if small or big timeranges have any affect