Ok got it - did you delete the security index as well? Can you run GET _cat/shards/.security*?v
here 
.security-7 0 p STARTED 88 195kb 10.138.136.163
.security-7 0 r STARTED 88 195kb 10.138.136.160
ok so we are ok with the security index and you did not specify the username/password previously.
So the result of the cluster allocation explain tells you that the restore failed:
"explanation" : "shard has failed to be restored from the snapshot [Backup_Repository:daily_backup--2022-09-13_11-00-00-evmwxc0usa-msh9csmgy2a/kvpVPTpxQryWjm8Z_fweGg] because of [failed shard on node [QZZkn1ASTsuKhhHvhOD03g]: failed recovery, failure RecoveryFailedException[[.kibana_security_session_1][0]: Recovery failed on {audynappld0013.corp.qbe.com}{QZZkn1ASTsuKhhHvhOD03g}{OynA96iQSy-Fl-dNk1R6pA}{10.138.136.161}{10.138.136.161:9303}{cdfhilmrstw}{ml.machine_memory=12409638912, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=2147483648}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [daily_backup--2022-09-13_11-00-00-evmwxc0usa-msh9csmgy2a/kvpVPTpxQryWjm8Z_fweGg]]; nested: SnapshotMissingException[[Backup_Repository:daily_backup--2022-09-13_11-00-00-evmwxc0usa-msh9csmgy2a/kvpVPTpxQryWjm8Z_fweGg] is missing]; nested: NoSuchFileException[/opt/elasticsearch/backupelastic/sys/indices/yOtUwKEdRtmClYRCFSO2ow/0/snap-kvpVPTpxQryWjm8Z_fweGg.dat]; ] - manually close or delete the index [.kibana_security_session_1] in order to retry to restore the snapshot again or use the reroute API to force the allocation of an empty primary shard"
I am not sure why it tells you that the snapshot Backup_Repository:daily_backup--2022-09-13_11-00-00-evmwxc0usa-msh9csmgy2a/kvpVPTpxQryWjm8Z_fweGg is missing.
Does this correspond to the snapshot you previously used?
yes, that's the name of my repository and snapshot I used
hmm Is there any way to recover the index?
Can you run GET _snapshot/repository_name/snapshot_name ? Replace the repository and snapshot names accordingly (with the one you used).
Well if the snapshot still exists and the data was in the snapshot, yes. You will need to the close that index and restore it from the snapshot. If the data is missing from the snapshot, then there is nothing we can do.
I could be mistaken but I think that index is being recreated by Kibana upon Kibana restart (if the index is missing). You may want to check that and restart Kibana.
For the record, this index .kibana_security_session_1 is not as critical as the kibana index - at it is only contains the saved login sessions. So losing it should have little impact.
as I checked, here is the message
{"error":{"root_cause":[{"type":"snapshot_missing_exception","reason":"[Backup_Repository:daily_backup--2022-09-13_11-00-00-evmwxc0usa-msh9csmgy2a] is missing"}],"type":"snapshot_missing_exception","reason":"[Backup_Repository:daily_backup--2022-09-13_11-00-00-evmwxc0usa-msh9csmgy2a] is missing"},"status":404}
is it possible to recreate the index?
Seems you lost the snapshot? I wonder if it is because you have an agressive SLM policy (to be checked...).
In the meantime, can you run DELETE .kibana_security_session_1 and restart Kibana?
but hmm, is it safe to delete the .kibana_security_session_1? Am I able to see this later on?
It is as per my comment above.
okay, I'll do delete then restart
btw, just wondering what would happen after I delete and restart kibana?
Kibana should recreate that index at the next restart/login.
1 Like
I see, done doing the steps. Just waiting for the Kibana response, it is in status of "Kibana server is not ready yet"
is it normal that the page is not responding immediately?
if everything is ok, you should be able to access the login page once Kibana is fully initialised, unless there are other errors - anything in the Kibana logs?
after I delete and restart, here is the logs says
{"type":"log","@timestamp":"2022-09-13T23:57:47+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:57:48+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:57:49+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 2 in 4 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:57:50+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 2 in 4 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:57:53+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 3 in 8 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:57:54+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 3 in 8 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:58:01+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:58:02+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:58:17+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 5 in 32 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:58:18+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 5 in 32 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:58:49+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 6 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:58:50+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 6 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:59:53+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 7 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-13T23:59:54+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 7 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:00:58+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 8 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:00:58+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 8 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:02:02+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9288486]: routing [null]]'. Retrying attempt 9 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:02:02+10:00","tags":["error","savedobjects-service"],"pid":62324,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2680790]: routing [null]]'. Retrying attempt 9 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:19+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9290225]: routing [null]]'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:20+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2682264]: routing [null]]'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:21+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9290225]: routing [null]]'. Retrying attempt 2 in 4 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:23+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2682264]: routing [null]]'. Retrying attempt 2 in 4 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:25+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9290225]: routing [null]]'. Retrying attempt 3 in 8 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:27+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2682264]: routing [null]]'. Retrying attempt 3 in 8 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:33+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9290225]: routing [null]]'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:35+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2682264]: routing [null]]'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:50+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9290225]: routing [null]]'. Retrying attempt 5 in 32 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:05:51+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2682264]: routing [null]]'. Retrying attempt 5 in 32 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:06:22+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana_task_manager] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][BsmpENn1R1aWOC23pyWc8Q:9290225]: routing [null]]'. Retrying attempt 6 in 64 seconds."}
{"type":"log","@timestamp":"2022-09-14T00:06:23+10:00","tags":["error","savedobjects-service"],"pid":64894,"message":"[.kibana] Action failed with 'no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.tasks][task][lxb0wEi-Sr665MEcO_7IGA:2682264]: routing [null]]'. Retrying attempt 6 in 64 seconds."}
The .kibana and .kibana_task_manager indices are allocated (version 7.16.2) (we have seen that before). So unless you tampered with the aliases or the indices, it should work.
just wondering, if the .kibana_security_session_1 can be mapped? in order to bring it back? hmm can we give it a try if possible? 
Can you share the results of GET _cat/aliases?v
1 Like