Kibana cannot find stack monitoring data when using Metricbeat to collect Elasticsearch/Kibana/Logstash metrics on 8.x
I have the below configurations:
Kibana (8.8.0)
No explicit settings in regards to monitoring etc.
Elasticsearch (8.8.1)
GET /_cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"elasticsearch": {
"collection": {
"enabled": "true"
}
},
"collection": {
"enabled": "false"
}
}
}
}
}
Metricbeat (8.8.1):
- module: kubernetes
metricsets:
- event
period: 10s
- module: beat
period: 10s
hosts: ["http://localhost:5066"]
xpack.enabled: true
- hosts:
- http://elasticsearch-client.default:9200
metricsets:
- cluster_stats
- node
- node_stats
- index
- index_recovery
- index_summary
- pending_stats
- shard
module: elasticsearch
period: 10s
scope: cluster
xpack.enabled: true
- enabled: true
hosts:
- http://kibana-ui.default:5601
module: kibana
period: 10s
xpack.enabled: true
(I have also tried emitting metricsets
entirely on the elasticsearch
module.
I can see metricbeat is writing data to the relevant monitoring indices as I would expect:
However, Kibana doesn't seem the monitoring data:
Error: Unable to find the cluster in the selected time range. UUID: 4JH7KXiPTd-yk6aXxJ5sxA
I've observed from Kibana's debug logs (monitoring.ui.debug_mode
) that Kibana is executing the below query when looking for monitoring data which is returning zero results:
POST *:.monitoring-es-*,.monitoring-es-*,*:metrics-elasticsearch.stack_monitoring.cluster_stats-*,metrics-elasticsearch.stack_monitoring.cluster_stats-*/_search
{
"query": {
"bool": {
"filter": [
{
"bool": {
"should": [
{
"term": {
"data_stream.dataset": "elasticsearch.stack_monitoring.cluster_stats"
}
},
{
"term": {
"metricset.name": "cluster_stats"
}
},
{
"term": {
"type": "cluster_stats"
}
}
]
}
}
]
}
},
"collapse": {
"field": "cluster_uuid"
},
"sort": {
"timestamp": {
"order": "desc",
"unmapped_type": "long"
}
}
}
Looking at the data in the .monitoring-es-8-mb
, none of the documents contain any of the fields in the filter query above.
If I set xpack.monitoring.collection.enabled
to true
I get monitoring data for Elasticsearch (but this is through the legacy collection method which I thought was actually deprecated in 8.x), but does not solve the issue for the Kibana, Logstash, Beats etc. stack monitoring