[SOLVED][ES 8.4.3] Monitoring metricbeat - Unable to find the cluster in the selected time range

Elastic Stack Elasticsearch
1

Hi,

I'm having a problem displaying monitoring on an Elasticsearch cluster (3 nodes), with the error message "Unable to find the cluster in the selected time range. UUID: 1xTxMsopToqNnvd_mbmcHw HTTP 404" when I visit the monitoring page on Kibana.

My cluster is composed by 3 Elasticsearch nodes in version 8.4.3 (3 pods in a Kubernetes cluster with master, data, ingest roles), 1 Kibana virtual machine in version 8.4.3 and 1 Metricbeat pods in version 8.4.3.

I followed the steps describes in the documentation Collecting Elasticsearch monitoring data with Metricbeat | Elasticsearch Guide [8.4] | Elastic to configure the monitoring of this cluster.

To further explain my cluster has the following settings

# GET _cluster/settings
{
  "persistent": {
    "xpack": {
      "monitoring": {
        "elasticsearch": {
          "collection": {
            "enabled": "false"
          }
        },
        "collection": {
          "enabled": "true"
        }
      }
    }
  },
  "transient": {}
}

Metricbeat is configured as follows:

#  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: true
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_LB}:9200']
      protocol: "https"
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.verification_mode: none

With modules configuration:

#  elasticsearch-xpack.yml: |-
    - module: elasticsearch
      metricsets:
        - cluster_stats
        - node
        - node_stats
        - index
        - index_recovery
        - index_summary
        - shard
        #- ml_job
        #- enrich
        #- ccr
      period: 10s
      scope: cluster
      hosts: ["https://${ELASTICSEARCH_LB}:9200"]
      username: ${ELASTICSEARCH_USERNAME}
      password:  ${ELASTICSEARCH_PASSWORD}
      ssl.verification_mode: "none"
      xpack.enabled: true

With ELASTICSEARCH_USERNAME is remote_monitoring_user.

I have no errors in the metricbeat pod logs.
I also have a data stream .monitoring-es-8-mb automatically created on my cluster, linked to a template index .monitoring-es-mb and ILM .monitoring-8-ilm-policy. An index named .ds-.monitoring-es-8-mb-2022.10.10-000001 is created with documents inside.

The documents contain the same UUID as my cluster

# GET .ds-.monitoring-es-8-mb-2022.10.10-000001/_search
# {"query":{"match_all":{}},"size":1,"sort":[{"@timestamp":{"order":"desc"}}]}
[ ... ]
    "hits": [
      {
        "_index": ".ds-.monitoring-es-8-mb-2022.10.10-000001",
        "_id": "2IO4woMBdYISlr49LFs-",
        "_score": null,
        "_source": {
          "@timestamp": "2022-10-10T16:26:32.375Z",
          "elasticsearch": {
            "cluster": {
              "id": "1xTxMsopToqNnvd_mbmcHw",
              "name": "mib-search"
            },
[ ... ]

# GET _cluster/state
{
  "cluster_name": "mib-search",
  "cluster_uuid": "1xTxMsopToqNnvd_mbmcHw",
  "version": 10414,
[ ... ]

When I visit the Kibana monitoring page (Tested with several timepick) I have the following error in Kibana logs :

[ERROR][plugins.monitoring.monitoring] Error: Unable to find the cluster in the selected time range. UUID: 1xTxMsopToqNnvd_mbmcHw
    at /usr/share/kibana/x-pack/plugins/monitoring/server/lib/cluster/get_cluster_stats.js:41:32
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.handler (/usr/share/kibana/x-pack/plugins/monitoring/server/routes/api/v1/elasticsearch/nodes.js:57:30)
    at handler (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:358:28)
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/router.js:163:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

I tried restarting the Elasticsearch cluster completely, as well as Kibana and Metricbeat.
Then I tried to delete the existing datasteam and monitoring indexes, but still the same problem.

kibana_monitoring_UUID_error
kibana_monitoring_UUID_error960×339 32.1 KB

What am I missing here?
Any ideas/suggestions would be much appreciated.

2

My problem came from missing ingest pipeline "xpack_monitoring_7" i think.
Once recreated I have the display of my monitoring data in Kibana ...

PUT _ingest/pipeline/xpack_monitoring_7
{
  "xpack_monitoring_7": {
    "description": "This is a placeholder pipeline for Monitoring API version 7 so that future versions may fix breaking changes.",
    "processors": [],
    "version": 7140099
  }
}

I hope this to help someone else.

3 Likes
3

Nice job finding that, @mib !

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.