Hi,
Due to the low disk space (95% used) on my system goes down. Then I have removed some files from the indices directory and restart both Kibana and Elasticsearch service.
But system is not working now.
According to the logs, I saw security index file is not available and then I try to reset password for both elastic and kibana_system user. But it's also not work for me. Can someone help me?
Thanks, in advanced.
rkulanga.
==> /var/log/elasticsearch/dev-pay-portal-es.log <==
[2022-09-21T05:08:10,196][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,237][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,317][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,785][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,963][ERROR][o.e.x.s.a.e.ReservedRealm] [ip-server-IP] failed to retrieve password hash for reserved user [kibana_system]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:147) ~[x-pack-security-7.16.1.jar:7.16.1]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:605) [x-pack-security-7.16.1.jar:7.16.1]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:231) [x-pack-security-7.16.1.jar:7.16.1]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:109) [x-pack-security-7.16.1.jar:7.16.1]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) [x-pack-security-7.16.1.jar:7.16.1]
[2022-09-21T05:08:10,965][INFO ][o.e.x.s.a.RealmsAuthenticator] [ip-server-IP] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2022-09-21T05:08:11,372][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:11,675][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:12,014][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
~# tail -f /var/log/kibana/kibana.log
{"type":"log","@timestamp":"2022-09-20T10:06:02+01:00","tags":["warning","plugins","reporting","config"],"pid":1323,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:02+01:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":1323,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:02+01:00","tags":["warning","plugins","actions"],"pid":1323,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:03+01:00","tags":["warning","plugins","alerting"],"pid":1323,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:03+01:00","tags":["info","plugins","ruleRegistry"],"pid":1323,"message":"Installing common resources shared between all indices"}
{"type":"log","@timestamp":"2022-09-20T10:06:03+01:00","tags":["info","plugins","reporting","config"],"pid":1323,"message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 18.04 OS. Automatically enabling Chromium sandbox."}
{"type":"log","@timestamp":"2022-09-20T10:06:04+01:00","tags":["warning","process"],"pid":1323,"message":"Error [ProductNotSupportedSecurityError]: The client is unable to verify that the server is Elasticsearch due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.\n at /usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:576:19\n at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:369:9)\n at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)\n at IncomingMessage.emit (node:events:402:35)\n at endReadableNT (node:internal/streams/readable:1343:12)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)"}
{"type":"log","@timestamp":"2022-09-20T10:06:06+01:00","tags":["error","elasticsearch-service"],"pid":1323,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
