Kibana missing credentials ver 9.0.3

Elastic Stack Kibana
1

Hello,

I receive errors in kibana logs

{"@timestamp":"2025-09-17T17:57:36.157Z","event":{"provider":"alerting","action":"execute","kind":"alert","category":\["monitoring"\],"start":"2025-09-17T17:57:36.121Z","outcome":"failure","end":"2025-09-17T17:
57:36.157Z","duration":"36000000","reason":"execute"},"kibana":{"alert":{"rule":{"rule_type_id":"monitoring_alert_missing_monitoring_data","consumer":"monitoring","execution":{"uuid":"d93eddf3-7661-4cf2-acdf-
63b96e8a600f","metrics":{"claim_to_start_duration_ms":19,"total_run_duration_ms":55,"prepare_rule_duration_ms":11,"rule_type_run_duration_ms":6,"process_alerts_duration_ms":0,"persist_alerts_duration_ms":0,"t
rigger_actions_duration_ms":0,"process_rule_duration_ms":11}}}},"saved_objects":\[{"rel":"primary","type":"alert","id":"beea88c5-bc35-4180-9f68-07f18b6151e6","type_id":"monitoring_alert_missing_monitoring_data
"}\],"space_ids":\["default"\],"task":{"scheduled":"2025-09-17T17:57:35.688Z","schedule_delay":433000000},"alerting":{"outcome":"failure","status":"error"},"server_uuid":"d9234d69-fee2-45a8-bf03-9803bbb6f872","v
ersion":"9.0.3"},"rule":{"id":"beea88c5-bc35-4180-9f68-07f18b6151e6","license":"basic","category":"monitoring_alert_missing_monitoring_data","ruleset":"monitoring","name":"Missing monitoring data"},"message":
"rule execution failure: monitoring_alert_missing_monitoring_data:beea88c5-bc35-4180-9f68-07f18b6151e6: 'Missing monitoring data'","error":{"message":"security_exception\\n\\tRoot causes:\\n\\t\\**tsecurity_exceptio**
**n: missing authentication credentials for REST request \[/*%3A.monitoring-es-*%2C.monitoring-es-*%2C*%3Ametrics-elasticsearch.stack_monitoring.cluster_stats-*%2Cmetrics-elasticsearch.stack_monitoring.cluster_s**
**tats-*/\_search?**filter_path=hits.hits.\_source.cluster_settings.cluster.metadata.display_name%2Chits.hits.\_source.cluster_uuid%2Chits.hits.\_source.elasticsearch.cluster.id%2Chits.hits.\_source.cluster_name%2Chit
s.hits.\_source.elasticsearch.cluster.name\]"},"ecs":{"version":"1.8.0"}

}

I use standard kibana_system user
below kibana.yml

server.host: "*******"
server.publicBaseUrl: "http://*******:5601/"
server.name: "elk-prod-logs"
elasticsearch.hosts: ["*******, *******, *******"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "*******"

logging.root.level: debug
logging.appenders.default:
  type: rolling-file
  fileName: /user/kibana-9.0.3/logs/kibana.log
  policy:
    type: size-limit
    size: 256mb
  strategy:
    type: numeric
    max: 10
  layout:
    type: pattern
    highlight: true


xpack.encryptedSavedObjects.encryptionKey: *
xpack.reporting.encryptionKey: *
xpack.security.encryptionKey: *

xpack.apm.ui.enabled: true
monitoring.ui.elasticsearch.username: "*******"
monitoring.ui.elasticsearch.password: "*******"

xpack.fleet.packages:
  - name: apm
    version: latest

xpack.fleet.isAirGapped: true
xpack.fleet.registryUrl: "http://*******:80"
2

Hello @Karol_Niema

Welcome to the community!!

Could you please try to curl & see what is the message :

curl -u kibana_system:<password> http://<es-host>:9200/

curl -u kibana_system:<password> "http://<es-host>:9200/.monitoring-es-*/_search?filter_path=hits.hits._source.cluster_name" -H 'Content-Type: application/json' -d '{"size":1}'

Thanks!!

3 

[****@******* ~]$ curl -u kibana_system:******** http://****:9200/
{
  "name" : "****",
  "cluster_name" : "elk-logging",
  "cluster_uuid" : "********",
  "version" : {
    "number" : "9.0.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "0a58bc1dc7a4ae5412db66624aab968370bd44ce",
    "build_date" : "2025-05-28T10:06:37.834829258Z",
    "build_snapshot" : false,
    "lucene_version" : "10.1.0",
    "minimum_wire_compatibility_version" : "8.18.0",
    "minimum_index_compatibility_version" : "8.0.0"
  },
  "tagline" : "You Know, for Search"
}

[****@**** ~]$ curl -u kibana_system:****** "http://*****:9200/.monitoring-es-*/_search?filter_path=hits.hits._source.cluster_name" -H 'Content-Type: application/json' -d '{"size":1}'

{
    "error": {
        "root_cause": [
            {
                "type": "security_exception",
                "reason": "unable to authenticate user [kibana_system] for REST request [/.monitoring-es-*/_search?filter_path=hits.hits._source.cluster_name]",
                "header": {
                    "WWW-Authenticate": [
                        "Basic realm=\"security\", charset=\"UTF-8\"",
                        "ApiKey"
                    ]
                }
            }
        ],
        "type": "security_exception",
        "reason": "unable to authenticate user [kibana_system] for REST request [/.monitoring-es-*/_search?filter_path=hits.hits._source.cluster_name]",
        "header": {
            "WWW-Authenticate": [
                "Basic realm=\"security\", charset=\"UTF-8\"",
                "ApiKey"
            ]
        }
    },
    "status": 401
}
4

Hello @Karol_Niema

From the logs it seems that the user does not have access to read the .monitoring indices. Still if you want to be sure instead of .monitoring can try .kibana* & if you receive the response than you need to create a new user with custom role as per your requirement. Post this try to query .monitoring-es indices.

Thanks!!

5

Problem occurs with default allerts, I deleted all allerts (no needed) and error does not exists

but
kibana use default role (kibana_system) and allerts was also built in, so it should work out of the box.