Stack Monitoring with Metricbeat 8 -- No Cluster Found

[Evesy (Michael Eves)]

Kibana cannot find stack monitoring data when using Metricbeat to collect Elasticsearch/Kibana/Logstash metrics on 8.x

I have the below configurations:

Kibana (8.8.0)
No explicit settings in regards to monitoring etc.

Elasticsearch (8.8.1)

GET /_cluster/settings
{
  "persistent": {
    "xpack": {
      "monitoring": {
        "elasticsearch": {
          "collection": {
            "enabled": "true"
          }
        },
        "collection": {
          "enabled": "false"
        }
      }
    }
  }
}

Metricbeat (8.8.1):

    - module: kubernetes
      metricsets:
        - event
      period: 10s
    - module: beat
      period: 10s
      hosts: ["http://localhost:5066"]
      xpack.enabled: true
    - hosts:
      - http://elasticsearch-client.default:9200
      metricsets:
      - cluster_stats
      - node
      - node_stats
      - index
      - index_recovery
      - index_summary
      - pending_stats
      - shard
      module: elasticsearch
      period: 10s
      scope: cluster
      xpack.enabled: true
    - enabled: true
      hosts:
      - http://kibana-ui.default:5601
      module: kibana
      period: 10s
      xpack.enabled: true

(I have also tried emitting metricsets entirely on the elasticsearch module.

I can see metricbeat is writing data to the relevant monitoring indices as I would expect:

However, Kibana doesn't seem the monitoring data:

Error: Unable to find the cluster in the selected time range. UUID: 4JH7KXiPTd-yk6aXxJ5sxA

I've observed from Kibana's debug logs (monitoring.ui.debug_mode) that Kibana is executing the below query when looking for monitoring data which is returning zero results:

POST *:.monitoring-es-*,.monitoring-es-*,*:metrics-elasticsearch.stack_monitoring.cluster_stats-*,metrics-elasticsearch.stack_monitoring.cluster_stats-*/_search
{
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                "term": {
                                    "data_stream.dataset": "elasticsearch.stack_monitoring.cluster_stats"
                                }
                            },
                            {
                                "term": {
                                    "metricset.name": "cluster_stats"
                                }
                            },
                            {
                                "term": {
                                    "type": "cluster_stats"
                                }
                            }
                        ]
                    }
                }
            ]
        }
    },
    "collapse": {
        "field": "cluster_uuid"
    },
    "sort": {
        "timestamp": {
            "order": "desc",
            "unmapped_type": "long"
        }
    }
}

Looking at the data in the .monitoring-es-8-mb, none of the documents contain any of the fields in the filter query above.

If I set xpack.monitoring.collection.enabled to true I get monitoring data for Elasticsearch (but this is through the legacy collection method which I thought was actually deprecated in 8.x), but does not solve the issue for the Kibana, Logstash, Beats etc. stack monitoring

[ramdas]

Thanks @Evesy for detailed debugging of the issue, i have been struggling with it too. however no replies from community experts yet.

Hi @warkolm @eMitch @ingri.mahecha, pinging you to get your kind attention on this issue. can you please help us fix this?

[eMitch (Eddie Mitchell)]

@Evesy - can you verify that at least one of the nodes in the cluster has the Ingest role assigned to it?

GET _cat/nodes?v

look for node.role and make sure you see an i

[Evesy (Michael Eves)]

Hi @eMitch -- Apologies for the delayed response. We have a number of ingest nodes in the cluster as you can see below:

10.178.5.4  18  98 27 4.10 3.98 4.43 hirs - elasticsearch-data-hot-4
10.178.14.4 44  97  1 0.54 0.97 0.88 irtw - elasticsearch-data-warm-1
10.178.6.3  71 100 34 5.75 4.12 3.73 hirs - elasticsearch-data-hot-2
10.178.0.9  64  83  6 0.87 0.67 0.42 m    * elasticsearch-master-2
10.178.13.4 33  99 47 4.99 4.29 4.26 hirs - elasticsearch-data-hot-1
10.178.7.4  51  96  0 0.06 0.14 0.29 irtw - elasticsearch-data-warm-4
10.178.3.16 40  82  8 1.07 0.79 0.58 m    - elasticsearch-master-0
10.178.8.3  23  97  1 0.21 0.24 0.41 irtw - elasticsearch-data-warm-2
10.178.2.3  47 100 23 3.22 3.69 4.04 hirs - elasticsearch-data-hot-3
10.178.11.3 30  98  1 0.54 0.63 0.68 irtw - elasticsearch-data-warm-3
10.178.9.3  57  97  1 0.14 0.20 0.42 irtw - elasticsearch-data-warm-0
10.178.4.4  20  90 32 6.13 6.16 5.51 hirs - elasticsearch-data-hot-0
10.178.10.8 17  82  1 0.22 0.35 0.40 m    - elasticsearch-master-1
10.178.12.4 46  96  1 0.30 0.31 0.45 irtw - elasticsearch-data-warm-6
10.178.15.4 58  94  1 0.02 0.11 0.37 irtw - elasticsearch-data-warm-5

Please let me know any other information I can provide

[ingri.mahecha (Ingri)]

Hello, verifying I do not see that you have configured the nodes, I leave you a reference URL Running Metricbeat on master nodes