Upgrading from 7.16.1 to 7.17.5 Broke Kibana Monitoring Tab

Earlier this week I upgraded from ES/Kibana 7.16.2 to 7.17.5. My setup consisted of a Production ES cluster and a separate single server as the monitoring cluster. The production cluster would push info to the monitoring cluster via exporter.

Symptoms: After upgrading I now get a screen asking if I'd like to setup metric beat or continue doing legacy. I started out by trying to do the legacy option, but it wasn't recognizing that anything was in the .monitoring-es-7-*. Kibana would also complain that the kibana node didn't have things like xpack.monitoring.collection.enabled: true. Which makes me think it wants to monitor itself and I don't want that.

Secondary Attempt: After beating my head with the legacy setup I decided to bread down and install metric beat on the production cluster (and the kibana node which I don't think I need). I set everything up so that the production cluster pushes elastic and system module data to the kibana node and I can see the data in a metricbeat-* index. I can even go into the dashboards section and view the graphs from the system module. However I want the data from the ES module.

After installing metric beat I went back into the monitoring tab of Kibana and tried navigating through the 'Setup Monitoring with Metric Beat'. The nodes from my production cluster don't show up. Clicking the 'Setup monitoring for a new node' and following all the steps is fine until I get to the bottom section 'Monitoring status' and the message 'No monitoring data detected, but we’ll continue checking.'

Quick tidbits about the system that I don't know where to put into this:

  • No security features enabled
  • The Kibana node and the Production cluster have different cluster UUIDs as expected

After you configured Metricbeat monitoring, did a new index show up on your monitoring cluster? it should be .monitoring-mb-es or something similar. if you have a metricbeat index it means that system monitoring is configured and you're just getting system monitoring data, not elastic stack monitoring data. You will need to go through all these steps:

I do see logs being logged to .monitoring-es-7-mb-{yyyy.MM.dd}. It does look like it's actual ES information and not just the system information as I saw both previously and have turned off the System module on my production cluster.

I may be getting confused, but I don't think I need metric beat on my monitoring node/cluster unless it is required to be able to read and display the data sent from the production cluster. That article looks like it's for monitoring Kibana itself.

Just to clarify I have a production cluster that is sending metric beat info logs to my monitoring cluster. This monitoring cluster is a single node running both ES and Kibana. I want Kibana's Stack Monitoring page to show the log information that is sent from the production cluster, but is stored on the monitoring node.