Elasticsearch log show kibana_version_mismatch

Elastic Stack Elasticsearch
1

Dears,

In elasticsearch log file there is many errors like this:

[2024-11-28T14:17:45,307][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [elastic03] failed to execute [search] input for watch [kfkGWL45QVSehYfLGLGl8g_kibana_version_mismatch], reason [all shards failed]
[2024-11-28T14:17:45,359][WARN ][o.e.x.w.e.ExecutionService] [elastic03] failed to execute watch [kfkGWL45QVSehYfLGLGl8g_kibana_version_mismatch]

ELK version: 7.17.25

What does it means? How to solve it? Any ideas what is wrong?

Best Regards,
Dan

2

There seems to be a version difference between Kibana and Elasticsearch. These should always be on the same version. Check exactly which versions you are running.

The version you are using is very old so you should look to upgrade to the latest version.

3

Kibana and elasticsearch versions are the same:

[root@elastic01 elasticsearch]# rpm -qa|grep elasticsearch
elasticsearch-7.17.25-1.x86_64
[root@elastic01 elasticsearch]#

[root@elastic02 elasticsearch]# rpm -qa|grep elasticsearch
elasticsearch-7.17.25-1.x86_64
[root@elastic02 elasticsearch]#

[root@elastic03 elasticsearch]# rpm -qa|grep elasticsearch
elasticsearch-7.17.25-1.x86_64
[root@elastic03 elasticsearch]#

[root@kibana01 ~]# rpm -qa|grep kibana
kibana-7.17.25-1.x86_64
[root@kibana01 ~]#

@Christian_Dahlqvist thanks for your response

4

What is the status of your cluster?

Can you share the result of the following request in dev tools?

GET _cluster/health

Do you have monitoring data in your cluster? The log you shared seems to be related to a built-in monitoring alert for Kibana Verison Mismatch.

it is not exactly saying that you have a version mismatch, but that it could not run the search because all shards failed.

5

@leandrojmp The cluster status is green:

{
  "cluster_name" : "elk01",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 1419,
  "active_shards" : 2838,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

It was the first thing what I've checked.

6

I checked last cluster logs from two last months and probably problem appears after last kibana upgrade from 7.17.23 too 7.17.25

7

That's weird.

Maybe some issue during upgrade?

What is the result of running this in Kibana Dev Tools.

GET kbn:/api/status
8

Is this right API method because:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [kbn:]",
        "resource.type" : "index_expression",
        "resource.id" : "kbn:",
        "index_uuid" : "_na_",
        "index" : "kbn:"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [kbn:]",
    "resource.type" : "index_expression",
    "resource.id" : "kbn:",
    "index_uuid" : "_na_",
    "index" : "kbn:"
  },
  "status" : 404
}
9

Yeah, it is right, but it seems that in version 7.X you cannot query Kibana API directly from Dev Tools.

I've not used version 7.X for the past years.

From the documentation you would need to query Kibana directly, but I'm not sure that this endpoint exists in 7.

This was just to show the Kibana version and if it was really updated, for the case in where the package was updated, but the running process is still running the older version.

10

Here you are the result kibana API:

{
    "name": "kibana01",
    "uuid": "7114b74c-059a-403f-b782-3b9b43964c31",
    "version": {
        "number": "7.17.25",
        "build_hash": "427e9521131a6f5f96fe79fb6d6eca013a5f89f3",
        "build_number": 47705,
        "build_snapshot": false
    },
    "status": {
        "overall": {
            "since": "2024-11-28T18:24:06.628Z",
            "state": "green",
            "title": "Green",
            "nickname": "Looking good",
            "icon": "success",
            "uiColor": "secondary"
        },
        "statuses": [
            {
                "id": "core:elasticsearch@7.17.25",
                "message": "Elasticsearch is available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "core:savedObjects@7.17.25",
                "message": "SavedObjects service has completed migrations and is available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:licensing@7.17.25",
                "message": "License fetched",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:banners@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:features@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:globalSearch@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:globalSearchProviders@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:uiActionsEnhanced@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:embeddableEnhanced@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:urlDrilldown@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:grokdebugger@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:painlessLab@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:searchprofiler@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:licenseManagement@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:watcher@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:spaces@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:discover@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:eventLog@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:graph@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:security@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visualizations@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:cloud@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:dashboard@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:dataEnhanced@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:discoverEnhanced@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:encryptedSavedObjects@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionMetricVis@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionTagcloud@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:fileUpload@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:ingestPipelines@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:lists@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:regionMap@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:reporting@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:savedObjectsTagging@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:tileMap@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeMarkdown@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeMetric@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypePie@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeTable@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeTagcloud@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeTimelion@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeVega@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeVislib@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeXy@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:actions@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:canvas@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:dashboardEnhanced@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:dashboardMode@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:fleet@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:globalSearchBar@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:lens@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:maps@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:savedObjectsManagement@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:snapshotRestore@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visualize@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:dataVisualizer@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:indexManagement@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visDefaultEditor@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:visTypeTimeseries@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:indexLifecycleManagement@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:inputControlVis@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:remoteClusters@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:rollup@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:triggersActionsUi@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:cases@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:crossClusterReplication@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:ml@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:osquery@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:ruleRegistry@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:stackAlerts@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:transform@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:timelines@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:observability@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:securitySolution@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:infra@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:uptime@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:apm@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:enterpriseSearch@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:monitoring@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:upgradeAssistant@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:logstash@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:alerting@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:bfetch@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:esUiShared@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressions@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:fieldFormats@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:kibanaLegacy@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:kibanaReact@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:kibanaUtils@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:mapsEms@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:screenshotMode@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:share@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:uiActions@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:usageCollection@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:code@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:licenseApiGuard@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:runtimeFields@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:translations@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:charts@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:dataViews@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:inspector@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:kibanaUsageCollection@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:mapsLegacy@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:newsfeed@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:telemetryCollectionManager@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:urlForwarding@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:xpackLegacy@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:data@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:devTools@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:embeddable@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:telemetry@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:telemetryCollectionXpack@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:indexPatternEditor@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:indexPatternFieldEditor@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:navigation@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:savedObjects@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:presentationUtil@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:savedObjectsTaggingOss@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:customIntegrations@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionError@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionImage@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionMetric@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionRepeatImage@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionRevealImage@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:expressionShape@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:home@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:console@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:kibanaOverview@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:management@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:advancedSettings@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:indexPatternManagement@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:telemetryManagementSection@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            },
            {
                "id": "plugin:taskManager@7.17.25",
                "message": "All dependencies are available",
                "since": "2024-11-28T18:24:06.628Z",
                "state": "green",
                "icon": "success",
                "uiColor": "secondary"
            }
        ]
    },
    "metrics": {
        "last_updated": "2024-11-28T18:24:04.424Z",
        "collection_interval_in_millis": 5000,
        "os": {
            "platform": "linux",
            "platformRelease": "linux-3.10.0-1160.125.1.el7.x86_64",
            "load": {
                "1m": 0.37,
                "5m": 0.46,
                "15m": 0.48
            },
            "memory": {
                "total_in_bytes": 16637321216,
                "free_in_bytes": 14869229568,
                "used_in_bytes": 1768091648
            },
            "uptime_in_millis": 2012909650,
            "distro": "Red Hat Linux",
            "distroRelease": "Red Hat Linux-7.9",
            "cpuacct": {
                "control_group": "/system.slice/kibana.service",
                "usage_nanos": 171245211770752
            },
            "cpu": {
                "control_group": "/system.slice/kibana.service",
                "cfs_period_micros": 100000,
                "cfs_quota_micros": -1,
                "stat": {
                    "number_of_elapsed_periods": 0,
                    "number_of_times_throttled": 0,
                    "time_throttled_nanos": 0
                }
            }
        },
        "process": {
            "memory": {
                "heap": {
                    "total_in_bytes": 315547648,
                    "used_in_bytes": 260746088,
                    "size_limit": 2197815296
                },
                "resident_set_size_in_bytes": 449708032
            },
            "pid": 8970,
            "event_loop_delay": 10.813280069414317,
            "event_loop_delay_histogram": {
                "min": 9.191424,
                "max": 54.296575,
                "mean": 10.813280069414317,
                "exceeds": 0,
                "stddev": 2.8659444813316997,
                "fromTimestamp": "2024-11-28T18:23:59.422Z",
                "lastUpdatedAt": "2024-11-28T18:24:04.421Z",
                "percentiles": {
                    "50": 10.199039,
                    "75": 10.403839,
                    "95": 13.139967,
                    "99": 22.937599
                }
            },
            "uptime_in_millis": 2011931859.2853081
        },
        "processes": [
            {
                "memory": {
                    "heap": {
                        "total_in_bytes": 315547648,
                        "used_in_bytes": 260746088,
                        "size_limit": 2197815296
                    },
                    "resident_set_size_in_bytes": 449708032
                },
                "pid": 8970,
                "event_loop_delay": 10.813280069414317,
                "event_loop_delay_histogram": {
                    "min": 9.191424,
                    "max": 54.296575,
                    "mean": 10.813280069414317,
                    "exceeds": 0,
                    "stddev": 2.8659444813316997,
                    "fromTimestamp": "2024-11-28T18:23:59.422Z",
                    "lastUpdatedAt": "2024-11-28T18:24:04.421Z",
                    "percentiles": {
                        "50": 10.199039,
                        "75": 10.403839,
                        "95": 13.139967,
                        "99": 22.937599
                    }
                },
                "uptime_in_millis": 2011931859.2853081
            }
        ],
        "response_times": {
            "avg_in_millis": 859.6,
            "max_in_millis": 1167
        },
        "concurrent_connections": 10,
        "requests": {
            "disconnects": 0,
            "total": 5,
            "statusCodes": {
                "200": 5
            },
            "status_codes": {
                "200": 5
            }
        }
    }
}
11

I don't think that the problem is necessarily in Kibana, or even that Elasticsearch and Kibana have a version mismatch. It looks like the problem is that the elastic-provided watch that tries to detect if Elasticsearch and Kibana have incompatible versions is failing to query the .monitoring-kibana-* indices for some reason. I would expect to see a stack trace in the elasticsearch logs though. My best guess (with no real evidence) is that maybe during an upgrade this watch became owned (and executed) by a user who doesn't have permission to see those indices. But hopefully the logs will tell us.

If you don't care about the results of the watch (or don't want to invest the time to diagnose the problem for this particular watch), you could always just deactivate the watch.

12

@Keith_Massey I know about possibility the deactivation of watcher but I think this is now solluction. Thanks