Security index is unavailable

Hi,
Due to the low disk space (95% used) on my system goes down. Then I have removed some files from the indices directory and restart both Kibana and Elasticsearch service.

But system is not working now.

According to the logs, I saw security index file is not available and then I try to reset password for both elastic and kibana_system user. But it's also not work for me. Can someone help me?
Thanks, in advanced.
rkulanga.

==> /var/log/elasticsearch/dev-pay-portal-es.log <==

[2022-09-21T05:08:10,196][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,237][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,317][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,785][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:10,963][ERROR][o.e.x.s.a.e.ReservedRealm] [ip-server-IP] failed to retrieve password hash for reserved user [kibana_system]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:147) ~[x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:605) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:231) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:109) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) [x-pack-security-7.16.1.jar:7.16.1]

[2022-09-21T05:08:10,965][INFO ][o.e.x.s.a.RealmsAuthenticator] [ip-server-IP] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2022-09-21T05:08:11,372][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:11,675][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]
[2022-09-21T05:08:12,014][ERROR][o.e.x.s.a.e.NativeUsersStore] [ip-server-IP] security index is unavailable. short circuiting retrieval of user [metricbeat_system]



~# tail -f /var/log/kibana/kibana.log 

{"type":"log","@timestamp":"2022-09-20T10:06:02+01:00","tags":["warning","plugins","reporting","config"],"pid":1323,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:02+01:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":1323,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:02+01:00","tags":["warning","plugins","actions"],"pid":1323,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:03+01:00","tags":["warning","plugins","alerting"],"pid":1323,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-09-20T10:06:03+01:00","tags":["info","plugins","ruleRegistry"],"pid":1323,"message":"Installing common resources shared between all indices"}
{"type":"log","@timestamp":"2022-09-20T10:06:03+01:00","tags":["info","plugins","reporting","config"],"pid":1323,"message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 18.04 OS. Automatically enabling Chromium sandbox."}
{"type":"log","@timestamp":"2022-09-20T10:06:04+01:00","tags":["warning","process"],"pid":1323,"message":"Error [ProductNotSupportedSecurityError]: The client is unable to verify that the server is Elasticsearch due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.\n    at /usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:576:19\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:369:9)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)\n    at IncomingMessage.emit (node:events:402:35)\n    at endReadableNT (node:internal/streams/readable:1343:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"}
{"type":"log","@timestamp":"2022-09-20T10:06:06+01:00","tags":["error","elasticsearch-service"],"pid":1323,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}

Welcome to our community! :smiley:

You didn't delete files manually did you? That's likely to be the problem for your issues if you did. You may need to try to recover a snapshot, or if you don't have one you can try elasticsearch-reset-password | Elasticsearch Guide [8.4] | Elastic, but you may have lost your data.

Thanks for the reply. Unfortunately, I delete the files manually. and I don't have a snapshot as well. however, I have followed the elasticsearch-reset-password | Elasticsearch Guide [8.4] | Elastic, But some command is not worked for me. Elasticsearch version is -7.16.1. Then I have created the restore super user and execute the below commands as well.

elasticsearch-users useradd restore_user -p password -r superuser

curl -u restore_user -k -X DELETE "https://elk-server:9200/.security-*"

I think my system is something ok. but system is not loading yet.

Browser output:
Kibana server is not ready yet

Logs:

org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_7.16.1_001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:239) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:466) [elasticsearch-7.16.1.jar:7.16.1]
	... 191 more

==> /var/log/elasticsearch/gc.log <==
[2022-09-21T07:16:25.868+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 1000183325 ns, Reaching safepoint: 118552 ns, At safepoint: 16157 ns, Total: 134709 ns
[2022-09-21T07:16:26.868+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 1000152684 ns, Reaching safepoint: 80399 ns, At safepoint: 7643 ns, Total: 88042 ns
[2022-09-21T07:16:29.868+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 3000419191 ns, Reaching safepoint: 116287 ns, At safepoint: 8167 ns, Total: 124454 ns


###########
tail -f /var/log/kibana/kibana.log 
{"type":"log","@timestamp":"2022-09-21T08:14:17+01:00","tags":["error","savedobjects-service"],"pid":9626,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 8 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-21T08:14:17+01:00","tags":["info","savedobjects-service"],"pid":9626,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64043ms."}
{"type":"log","@timestamp":"2022-09-21T08:15:21+01:00","tags":["error","savedobjects-service"],"pid":9626,"message":"[.kibana] Action failed with 'search_phase_execution_exception: '. Retrying attempt 9 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-21T08:15:21+01:00","tags":["info","savedobjects-service"],"pid":9626,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64079ms."}

Please help me, Thanks

We'd need to see more of your Elasticsearch log.

Given that Elasticsearch performs consistency checks on the data on disk and you manually deleted data I suspect this may mean that you have lost your data (as you have no valid snapshot to restore from).

If you run out of disk space the correct cource of action is to delete indices through the Elasticsearch APIs. You must NEVER EVER make changes at the file system level.

1 Like

	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:485) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:303) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:91) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:39) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:577) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:85) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:103) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:571) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:556) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:450) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:967) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:931) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:352) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1015) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:343) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:443) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:371) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:256) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:138) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:277) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:144) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:127) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:258) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$0(RestOpenPointInTimeAction.java:45) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:90) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:85) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:376) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:352) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:205) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:128) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$3(AuthenticatorChain.java:175) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:219) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:164) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:148) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:84) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:171) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:143) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:104) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) [x-pack-security-7.16.1.jar:7.16.1]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:327) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:393) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:245) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:32) [transport-netty4-client-7.16.1.jar:7.16.1]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:18) [transport-netty4-client-7.16.1.jar:7.16.1]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.16.1.jar:7.16.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_7.16.1_001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:239) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:466) [elasticsearch-7.16.1.jar:7.16.1]
	... 191 more

==> /var/log/elasticsearch/gc.log <==
[2022-09-21T07:29:05.700+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 11001329390 ns, Reaching safepoint: 81184 ns, At safepoint: 8305 ns, Total: 89489 ns
[2022-09-21T07:29:07.700+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 2000319733 ns, Reaching safepoint: 87197 ns, At safepoint: 8725 ns, Total: 95922 ns
[2022-09-21T07:29:14.701+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 7000892856 ns, Reaching safepoint: 86694 ns, At safepoint: 7391 ns, Total: 94085 ns
[2022-09-21T07:29:15.702+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 1000170061 ns, Reaching safepoint: 103155 ns, At safepoint: 6916 ns, Total: 110071 ns
[2022-09-21T07:29:22.703+0000][1018][safepoint   ] Safepoint "Cleanup", Time since last: 7000906336 ns, Reaching safepoint: 115905 ns, At safepoint: 7779 ns, Total: 123684 ns

Noted with Thanks.

Kibana.log

tail -f /var/log/kibana/kibana.log 
{"type":"log","@timestamp":"2022-09-21T09:14:45+01:00","tags":["error","savedobjects-service"],"pid":20156,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 6 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-21T09:14:45+01:00","tags":["info","savedobjects-service"],"pid":20156,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 32040ms."}
{"type":"log","@timestamp":"2022-09-21T09:15:49+01:00","tags":["error","savedobjects-service"],"pid":20156,"message":"[.kibana] Action failed with 'search_phase_execution_exception: '. Retrying attempt 7 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-21T09:15:49+01:00","tags":["info","savedobjects-service"],"pid":20156,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64037ms."}
{"type":"log","@timestamp":"2022-09-21T09:15:49+01:00","tags":["error","savedobjects-service"],"pid":20156,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 7 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-21T09:15:49+01:00","tags":["info","savedobjects-service"],"pid":20156,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64039ms."}
{"type":"log","@timestamp":"2022-09-21T09:16:53+01:00","tags":["error","savedobjects-service"],"pid":20156,"message":"[.kibana] Action failed with 'search_phase_execution_exception: '. Retrying attempt 8 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-21T09:16:53+01:00","tags":["info","savedobjects-service"],"pid":20156,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64040ms."}

You might need to delete that index and restart Kibana.

Thanks, Warkolm.
Can I use below command to delete index? Or Do I need to provide index pattern? If yes, how do I found it? I am a newbie for Kibana /Elasticsearch.
curl -u restore_user -k -X DELETE "https://elk-server:9200/index-pattern"

Thanks again.

That should work, yes.

I think, I need to provide the index pattern. otherwise, it will generate below error. No idea about that? please help. Thanks
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [index-pattern]","resource.type":"index_or_alias","resource.id":"index-pattern","index_uuid":"na","index":"index-pattern"}],"type":"index_not_found_exception","reason":"no such index [index-pattern]","resource.type":"index_or_alias","resource.id":"index-pattern","index_uuid":"na","index":"index-pattern"},"status":404}

You need to change index-pattern to the actual index name you want to delete, in this case .kibana_7.16.1_001.