Hi,
I'm having a problem displaying monitoring on an Elasticsearch cluster (3 nodes), with the error message "Unable to find the cluster in the selected time range. UUID: 1xTxMsopToqNnvd_mbmcHw HTTP 404" when I visit the monitoring page on Kibana.
My cluster is composed by 3 Elasticsearch nodes in version 8.4.3 (3 pods in a Kubernetes cluster with master, data, ingest roles), 1 Kibana virtual machine in version 8.4.3 and 1 Metricbeat pods in version 8.4.3.
I followed the steps describes in the documentation Collecting Elasticsearch monitoring data with Metricbeat | Elasticsearch Guide [8.4] | Elastic to configure the monitoring of this cluster.
To further explain my cluster has the following settings
# GET _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"elasticsearch": {
"collection": {
"enabled": "false"
}
},
"collection": {
"enabled": "true"
}
}
}
},
"transient": {}
}
Metricbeat is configured as follows:
# metricbeat.yml: |-
metricbeat.config.modules:
# Mounted `metricbeat-daemonset-modules` configmap:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: true
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_LB}:9200']
protocol: "https"
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl.verification_mode: none
With modules configuration:
# elasticsearch-xpack.yml: |-
- module: elasticsearch
metricsets:
- cluster_stats
- node
- node_stats
- index
- index_recovery
- index_summary
- shard
#- ml_job
#- enrich
#- ccr
period: 10s
scope: cluster
hosts: ["https://${ELASTICSEARCH_LB}:9200"]
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl.verification_mode: "none"
xpack.enabled: true
With ELASTICSEARCH_USERNAME is remote_monitoring_user.
I have no errors in the metricbeat pod logs.
I also have a data stream .monitoring-es-8-mb automatically created on my cluster, linked to a template index .monitoring-es-mb and ILM .monitoring-8-ilm-policy. An index named .ds-.monitoring-es-8-mb-2022.10.10-000001 is created with documents inside.
The documents contain the same UUID as my cluster
# GET .ds-.monitoring-es-8-mb-2022.10.10-000001/_search
# {"query":{"match_all":{}},"size":1,"sort":[{"@timestamp":{"order":"desc"}}]}
[ ... ]
"hits": [
{
"_index": ".ds-.monitoring-es-8-mb-2022.10.10-000001",
"_id": "2IO4woMBdYISlr49LFs-",
"_score": null,
"_source": {
"@timestamp": "2022-10-10T16:26:32.375Z",
"elasticsearch": {
"cluster": {
"id": "1xTxMsopToqNnvd_mbmcHw",
"name": "mib-search"
},
[ ... ]
# GET _cluster/state
{
"cluster_name": "mib-search",
"cluster_uuid": "1xTxMsopToqNnvd_mbmcHw",
"version": 10414,
[ ... ]
When I visit the Kibana monitoring page (Tested with several timepick) I have the following error in Kibana logs :
[ERROR][plugins.monitoring.monitoring] Error: Unable to find the cluster in the selected time range. UUID: 1xTxMsopToqNnvd_mbmcHw
at /usr/share/kibana/x-pack/plugins/monitoring/server/lib/cluster/get_cluster_stats.js:41:32
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.handler (/usr/share/kibana/x-pack/plugins/monitoring/server/routes/api/v1/elasticsearch/nodes.js:57:30)
at handler (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:358:28)
at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/router.js:163:30)
at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/router.js:124:50)
at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
I tried restarting the Elasticsearch cluster completely, as well as Kibana and Metricbeat.
Then I tried to delete the existing datasteam and monitoring indexes, but still the same problem.
What am I missing here?
Any ideas/suggestions would be much appreciated.