Nodes fail to list

I have Kibana 7.1 and after adding 10 more ES data nodes for a total of 30, the monitoring nodes tab is not listing any nodes. Any ideas as to what might have happened. Listing nodes using the ES API works fine.

Hi @o1o1o11o1,

thanks for your question. Are there any errors shown in the browser console?

Hi @o1o1o11o1,

Let's see what's going on here. Would you mind running the following Elasticsearch query against your monitoring Elasticsearch cluster and posting the results here?

{
  "_source": [
    "timestamp",
    "source_node",
    "node_stats.node_id"
  ],
  "query": {
    "term": {
      "type": {
        "value": "node_stats"
      }
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

Thanks,

Shaunak

I didn't see any errors, but changing the time frame from 1 hour to 15 minutes shows the nodes.

Based on this, could you amend the query I posted earlier slightly as shown below and re-run it, please?

POST .monitoring-es-*/_search
{
  "_source": [
    "timestamp",
    "source_node",
    "node_stats.node_id"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "type": {
              "value": "node_stats"
            }
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-1h/m",
              "lte": "now/m"
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

the above query returns 10 out of the 30 nodes.

Would you mind posting the complete results of the query? Or if not, could you at least post the values of the timestamp field for each of the 10 nodes?

Finally, could you re-run the same query with "gte": "now-15m/m" instead of "gte": "now-1h/m" and post those results as well?

Thanks,

Shaunak