Kibana Task Manager (KTM) not reporting accurate observed Kibana instances values

[JGreene (Jared Greene)]

Hello,

Looking for guidance on troubleshooting Kibana Task Manager reporting
"observed":{"observed_kibana_instances":1 when there are 12 instances in total. Other clusters (7.16.X) report their total Kibana instances accurately, but the 4 clusters upgraded to 8.9.0 no longer report accurate observed Kibana instances values.

We have a KTM dashboard and alerts where we rely on this data for scaling.

All of the Kibana instances are green on their status page and Rules/Alerts seem to be processing fine. The only other observation is that when on 7.16, out time to drain fluctuated between 1 & 2, currently all of the

The Kibana logs do not report anything related to an error with comms or anything else. Cluster/Kibana seems to be fine in all respects.

{
    "status": "OK",
    "timestamp": "2023-07-30T16:11:11.942Z",
    "value": {
        "observed": {
            "observed_kibana_instances": 1,     <<< Actual number is 12
            "max_throughput_per_minute_per_kibana": 2000,
            "max_throughput_per_minute": 2000,
            "minutes_to_drain_overdue": 6,
            "avg_recurring_required_throughput_per_minute": 182,
            "avg_recurring_required_throughput_per_minute_per_kibana": 182,
            "avg_required_throughput_per_minute": 182,
            "avg_required_throughput_per_minute_per_kibana": 182
        },
        "proposed": {
            "provisioned_kibana": 1,
            "min_required_kibana": 1,
            "avg_recurring_required_throughput_per_minute_per_kibana": 182,
            "avg_required_throughput_per_minute_per_kibana": 182
        }
    }
}

Any guidance would be much appreciated.