# Help restoring / recreating .security-7 **URL:** \ **Category:** Kibana\ **Created:** [May 27, 2023, 1:16pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499 "2023-05-27T13:16:43Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 1:16pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/1 "2023-05-27T13:16:43Z")
Hello, Faced an error where kibana failed to retrieve password hash for reserved user [kibana] at least one primary shard for the index .security-7 was missing. Followed the instructions provided here at [https://discuss.elastic.co/t/elk-7-8-0-two-node-cluster-at-least-one-primary-shard-for-the-index-security-7-is-unavailable/261031/3](https://discuss.elastic.co/t/elk-7-8-0-two-node-cluster-at-least-one-primary-shard-for-the-index-security-7-is-unavailable/261031/3), but still not able to authenticate. curl -XGET '[http://localhost:9200/\_cat/indices?v](http://localhost:9200/_cat/indices?v)' print the following: ```auto health status index uuid pri rep docs.count docs.deleted store.size pri.store.size red open .siem-signals-default-000001 BhEluMC5R3O-1UAOEXNjsA 1 1 red open .siem-signals-default-000002 3KWe-ir6RYqleiCzpjuYJg 1 1 red open .items-default-000001 I4W5BpmmRAmENBj1gaWgTQ 1 1 yellow open cf_http-2023-05-12 BzfKRxqXQNuydXlP29uf0A 1 1 11283152 0 30.4gb 30.4gb yellow open cf_http-2023-05-13 KkRwYjzTTV210AW97OrZHA 1 1 10963917 0 29.6gb 29.6gb yellow open cf_http-2023-05-14 38gVUUdiTSasH-0brxKeAg 1 1 10100763 0 27.2gb 27.2gb yellow open cf_http-2023-05-15 9WwqG5IWSwShflS-eXYiMg 1 1 9671741 0 26.1gb 26.1gb yellow open cf_http-2023-05-10 9e_Sg58FTXazTU8tTUCKZQ 1 1 14457458 0 37.3gb 37.3gb yellow open cf_http-2023-05-11 WyZLo7OxTM2sjwBZSVkP_g 1 1 11294386 0 30.5gb 30.5gb yellow open cf_http-2023-05-16 ZbZmILzUQwCjB_p6hgXa_g 1 1 9532518 0 25.6gb 25.6gb yellow open cf_http-2023-05-17 QOzmmB_TQF-pOZbJfcu-Eg 1 1 9441068 0 25.4gb 25.4gb yellow open cf_http-2023-05-18 UStSBmDHRmiUj8kWPwbVlQ 1 1 9350414 0 25gb 25gb yellow open cf_http-2023-05-19 4BUXtylyTkum_QO3gyEb9Q 1 1 9885463 0 26.5gb 26.5gb green open .monitoring-es-7-2023.05.20 6XMbXQYJSSa9UjDdVE27mQ 1 0 537029 375300 299.7mb 299.7mb green open .monitoring-es-7-2023.05.21 CN2by8TfRz69SWKHqWCkEg 1 0 545881 393954 309.5mb 309.5mb green open .monitoring-es-7-2023.05.24 HCx8HYl7RTqkS0EzZaNDww 1 0 571985 33810 304.1mb 304.1mb green open .monitoring-es-7-2023.05.25 vXKUAm40Rq6KtbL6TaS1Yw 1 0 580276 51554 308.7mb 308.7mb green open .monitoring-es-7-2023.05.22 kIDgCjWsTnyAsnk2Yg8hEQ 1 0 554392 412126 315.2mb 315.2mb green open .monitoring-es-7-2023.05.23 FYbchlz6Qjm7BbYWjLjebg 1 0 562834 16675 298mb 298mb green open .monitoring-es-7-2023.05.26 pDTO7vzURoqvKW4-Sp5o9Q 1 0 587087 64930 314.2mb 314.2mb green open .monitoring-es-7-2023.05.27 yEX548AlTMarN27IYqlv0Q 1 0 200350 23698 54.1mb 54.1mb red open cf_http-2023-05-01 rsBQcEr8QWmAUXHCxvLMNQ 1 1 yellow open cf_http-2023-05-02 WajoPx6hSY2dv7STRN8LtA 1 1 13920382 0 31.3gb 31.3gb red open cf_http-2023-04-30 jWwRcnJmS2quk_Q7ojoqcg 1 1 red open cf_http-2023-05-03 klBSHCWyQG-wFozFDdUFag 1 1 red open cf_http-2023-05-04 pzKrRRHpTguMdYfcWuObVg 1 1 yellow open cf_http-2023-05-09 Op-6odsZQdS-vVckZxnMuA 1 1 2010508 0 4.5gb 4.5gb red open cf_http-2023-05-05 o3ByeljATImygRIkpbnSHw 1 1 red open .fleet-policies-7 2r0eUwo4TSGwlbgyCWarTw 1 0 red open cf_http-2023-05-06 bLgNcFHYSH-Vj7y66-DbxA 1 1 red open cf_http-2023-05-07 y3tQQmWiRO6E5Enigjq2TQ 1 1 red open .metrics-endpoint.metadata_united_default BGe68IHwQPqa-s6kil70vQ 1 0 yellow open cf_http-2023-05-08 ZjBvR0EJR2GkQS0acRdMXw 1 1 11685460 0 26.2gb 26.2gb green open .kibana_task_manager_7.17.2_001 Lp2MDQmaQbK1dmytD4ndaA 1 0 18 525 102.8kb 102.8kb red open .kibana_7.17.2_001 aIleKa3GSwa6TEK1xDrgSA 1 0 yellow open cf_http-2023-04-21 -EiHodLuSvOYBmUEA9mSXg 1 1 68328067 0 154.8gb 154.8gb red open cf_http-2023-04-20 62ElZXxvTcKSfjpE5WDvtg 1 1 yellow open cf_http-2023-04-24 Ub-fxfHWQ22caIg_fa2G_g 1 1 25035372 0 56.7gb 56.7gb yellow open cf_http-2023-04-28 VBBgL13ZQPWsIl0CNpYHXg 1 1 19507590 0 44gb 44gb green open .transform-internal-007 XyT6oWZPQUau0Qxxn9O93w 1 0 12 7 78kb 78kb red open .lists-default-000001 42W-XczQTf2z2eLNwL3m5g 1 1 green open .monitoring-kibana-7-2023.05.23 gVDqg65RSUWbl9JaCU8Scw 1 0 17280 0 3.6mb 3.6mb red open metrics-endpoint.metadata_current_default IgOvTNOAQNqTwXFbUpNc_A 1 0 yellow open cf_http-2023-05-23 QaQo2iN4TKSrpA76eGcG6Q 1 1 37516121 0 102.6gb 102.6gb green open .monitoring-kibana-7-2023.05.22 6TCsRhQ2RGefmMW18Xm0UQ 1 0 17280 0 3.6mb 3.6mb yellow open cf_http-2023-05-24 oJMH9FRCRDO1ucIH_AEMqg 1 1 60072553 0 163.5gb 163.5gb green open .monitoring-kibana-7-2023.05.21 XUtgdhrhSoWcaHnVrnC32Q 1 0 17278 0 3.5mb 3.5mb yellow open cf_http-2023-05-25 Wjn87cQbTGG3spTXlL54cg 1 1 44819503 0 122.7gb 122.7gb green open .monitoring-kibana-7-2023.05.20 D555wOfsSBeq9HToP8khPQ 1 0 17278 0 3.3mb 3.3mb yellow open cf_http-2023-05-26 yWffxAHaQeGfCEhwe1kbKw 1 1 46868527 0 128.3gb 128.3gb yellow open cf_http-2023-05-20 bKBO_MdSSOKOHwyWF3VP1A 1 1 9730534 0 26.1gb 26.1gb yellow open cf_http-2023-05-21 EZkwZ04zQGO-CRHH2BeiCA 1 1 38507468 0 105.7gb 105.7gb yellow open cf_http-2023-05-22 R71k7MDYRC6E-ER9weV6bQ 1 1 48289529 0 132gb 132gb yellow open cf_http-2023-04-15 NJV0MX9oT8GNZv8dWAXiLw 1 1 59439724 0 134.9gb 134.9gb green open .monitoring-kibana-7-2023.05.27 4nLHtZr7TOilskvOcpLy8Q 1 0 4318 0 1mb 1mb green open .monitoring-kibana-7-2023.05.26 -OS_nWdCQvugQcqln5l1ww 1 0 17222 0 3.6mb 3.6mb green open .monitoring-kibana-7-2023.05.25 xJstPueGS629odwCuRqt2A 1 0 17280 0 3.7mb 3.7mb green open .monitoring-kibana-7-2023.05.24 IS17YSbnTXy3TgpqZiZ0sA 1 0 17278 0 3.5mb 3.5mb ``` restarting kibana brings that error: ```auto [ERROR][plugins.alerting.monitoring_alert_disk_usage] Executing Rule default:monitoring_alert_disk_usage:a6ce4d60-eedb-11ed-8583-45292c39ddfa has resulted in Error: security_exception: [security_exception] Reason: unable to authenticate with provided credentials and anonymous access is not allowed for this request, caused by: "" - Error: security_exception: [security_exception] Reason: unable to authenticate with provided credentials and anonymous access is not allowed for this request ``` Restarting Elasticsearch brings that error ```auto o.e.x.s.a.RealmsAuthenticator] [ELASTICSEARCH-NODE-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic] ``` Tried to reset Elastic and Kibana passwords but no luck. Any idea what is a possible cause for that? ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 4:34pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/2 "2023-05-27T16:34:05Z")
Here's another error that i just caught from Kibanalogs unavailable\_shards\_exception: [unavailable\_shards\_exception] Reason: at least one primary shard for the index [.security-profile-8] is unavailable ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 27, 2023, 5:06pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/3 "2023-05-27T17:06:36Z")
Most likely your disk is full please run the following `curl http://localhost:9200/_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r` and `curl http://localhost:9200` ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 5:19pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/4 "2023-05-27T17:19:55Z")
First request returns: ```auto ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 7 71 0 0.03 0.14 0.13 cdfhilmrstw * ELASTICSEARCH-NODE-1 ``` Second one returns: ```auto { "name" : "ELASTICSEARCH-NODE-1", "cluster_name" : "elasticsearch", "cluster_uuid" : "2ZQOMO8wSei1wJxjzL_8VA", "version" : { "number" : "8.6.2", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29", "build_date" : "2023-02-13T09:35:20.314882762Z", "build_snapshot" : false, "lucene_version" : "9.4.2", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } ``` ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 5:24pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/5 "2023-05-27T17:24:18Z")
I'm suspecting the absence of .security-7 system index ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 27, 2023, 5:35pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/6 "2023-05-27T17:35:02Z")
> [@stephenb](#): > > curl [http://localhost:9200/\_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r](http://localhost:9200/_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r) I don't think you ran that command because it's not showing the disk usage and disk total and disk percent which is a thing we need to look at. You just ran the cat nodes without the rest of it ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 5:46pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/7 "2023-05-27T17:46:58Z")
![elk2](https://us1.discourse-cdn.com/elastic/original/3X/7/b/7bc04edba59b4da04dffc99bd7e281411c02f47d.png) Actually, I executed the full command ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 27, 2023, 6:05pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/8 "2023-05-27T18:05:01Z")
Try to paste it in again or cut it in again.... You can see there's an underline where it thinks the link ends... You can go look up that in the documents we need to see the disk space Most likely you need to delete an index... Or provide more disk space ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 6:10pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/9 "2023-05-27T18:10:47Z")
I think this should answer your question ```auto shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 56 1.6tb 1.6tb 1.3tb 2.9tb 55 localhost 127.0.0.1 ELASTICSEARCH-NODE-1 77 UNASSIGNED ``` ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 27, 2023, 6:15pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/10 "2023-05-27T18:15:58Z")
I have over 1 TB available disk space. I there a way to rebuild the .security-7 index other than restoring a snapshot? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 27, 2023, 9:04pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/11 "2023-05-27T21:04:48Z")
I changed the Title to something more accurate/ helpful. Restoring is the best approach... Perhaps with the better title you might get more suggestions. It is not clear to me how to recreate the index... Did you try to reset the elastic password? > **[elasticsearch-reset-password | Elasticsearch Guide \[8.8\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/reset-password.html)** Also did you stop and start elasticsearch? Then try resetting the elastic password This is the danger of a single node cluster... It looks like you have lost other indices as well... do you have any idea how you lost it in the first place? ---
**Author:** ![A.Hani](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Post date:** [May 28, 2023, 6:36am UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/12 "2023-05-28T06:36:03Z")
Hi stephenb, I managed to restore the index by upgrading Elasticsearch and Kibana. Now the above errors are gone and only kibana is still throwing that error ```auto [ERROR][savedobjects-service] [.kibana] Action `failed with 'no_shard_available_action_ex> May 28 06:30:25 ELASTICSEARCH-NODE-1 kibana[882706]: Root causes: no_shard_available_action_exception ``` checking the status of the shards i can see that there still UNASSIGNED PRIMARY shards, reason: CLUSTER\_RECOVERED ```auto .geoip_databases 0 p UNASSIGNED CLUSTER_RECOVERED .tasks 0 p UNASSIGNED CLUSTER_RECOVERED .kibana_security_session_1 0 p UNASSIGNED CLUSTER_RECOVERED .kibana-event-log-7.17.2-000011 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-12 0 r UNASSIGNED CLUSTER_RECOVERED .reporting-2023-04-23 0 p UNASSIGNED CLUSTER_RECOVERED .reporting-2022-06-19 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-25 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-06 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-06 0 r UNASSIGNED CLUSTER_RECOVERED .reporting-2023-03-12 0 p UNASSIGNED CLUSTER_RECOVERED .fleet-policies-7 0 p UNASSIGNED CLUSTER_RECOVERED .siem-signals-default-000001 0 p UNASSIGNED CLUSTER_RECOVERED .siem-signals-default-000001 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-19 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-20 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-20 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-24 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-24 0 r UNASSIGNED CLUSTER_RECOVERED .kibana-event-log-7.17.2-000013 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-05 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-05 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-16 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-19 0 r UNASSIGNED CLUSTER_RECOVERED .items-default-000001 0 p UNASSIGNED CLUSTER_RECOVERED .items-default-000001 0 r UNASSIGNED CLUSTER_RECOVERED .async-search 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-07 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-07 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-14 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-10 0 r UNASSIGNED CLUSTER_RECOVERED .reporting-2022-04-24 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-04 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-04 0 r UNASSIGNED CLUSTER_RECOVERED .transform-notifications-000002 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-18 0 r UNASSIGNED CLUSTER_RECOVERED .security-profile-8 0 p UNASSIGNED CLUSTER_RECOVERED .metrics-endpoint.metadata_united_default 0 p UNASSIGNED CLUSTER_RECOVERED .ds-.slm-history-5-2023.04.28-000001 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-02 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-13 0 r UNASSIGNED CLUSTER_RECOVERED .ds-.logs-deprecation.elasticsearch-default-2023.03.29-000023 0 p UNASSIGNED CLUSTER_RECOVERED .reporting-2023-03-19 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-01 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-01 0 r UNASSIGNED CLUSTER_RECOVERED .lists-default-000001 0 p UNASSIGNED CLUSTER_RECOVERED .lists-default-000001 0 r UNASSIGNED CLUSTER_RECOVERED .fleet-enrollment-api-keys-7 0 p UNASSIGNED CLUSTER_RECOVERED .kibana_7.17.2_001 0 p UNASSIGNED CLUSTER_RECOVERED .ds-.logs-deprecation.elasticsearch-default-2023.04.28-000025 0 p UNASSIGNED CLUSTER_RECOVERED .apm-agent-configuration 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-03 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-03 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-21 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-15 0 r UNASSIGNED CLUSTER_RECOVERED metrics-endpoint.metadata_current_default 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-22 0 r UNASSIGNED CLUSTER_RECOVERED .reporting-2023-01-29 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-26 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-30 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-30 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-23 0 r UNASSIGNED CLUSTER_RECOVERED .reporting-2023-02-05 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-17 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-09 0 r UNASSIGNED CLUSTER_RECOVERED .apm-custom-link 0 p UNASSIGNED CLUSTER_RECOVERED .reporting-2023-02-26 0 p UNASSIGNED CLUSTER_RECOVERED .ds-ilm-history-5-2023.01.28-000018 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-04-28 0 r UNASSIGNED CLUSTER_RECOVERED .siem-signals-default-000002 0 p UNASSIGNED CLUSTER_RECOVERED .siem-signals-default-000002 0 r UNASSIGNED CLUSTER_RECOVERED .reporting-2022-12-18 0 p UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-11 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-20 0 r UNASSIGNED CLUSTER_RECOVERED cf_http-2023-05-08 0 r UNASSIGNED CLUSTER_RECOVERED .kibana_alerting_cases_8.8.0_reindex_temp 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.27 0 p STARTED ELASTICSEARCH-NODE-1 .reporting-2022-06-26 0 p STARTED ELASTICSEARCH-NODE-1 .reporting-2022-08-14 0 p STARTED ELASTICSEARCH-NODE-1 .kibana-event-log-7.17.2-000012 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.23 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.26 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_ingest_8.8.0_reindex_temp 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.27 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-12 0 p STARTED ELASTICSEARCH-NODE-1 .security-7 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-25 0 p STARTED ELASTICSEARCH-NODE-1 .reporting-2023-03-05 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_ingest_8.8.0_001 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_analytics_8.8.0_reindex_temp 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-19 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_security_solution_8.8.0_001 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-04-24 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-24 0 p STARTED ELASTICSEARCH-NODE-1 .kibana-event-log-8.6.2-000001 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-16 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.22 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_task_manager_8.6.2_001 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.24 0 p STARTED ELASTICSEARCH-NODE-1 .reporting-2023-04-02 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-04-19 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-14 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.28 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-10 0 p STARTED ELASTICSEARCH-NODE-1 .kibana-event-log-7.17.2-000014 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.22 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-18 0 p STARTED ELASTICSEARCH-NODE-1 .ds-ilm-history-5-2023.02.27-000020 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-02 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-13 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_8.8.0_001 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.25 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.26 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_task_manager_7.17.2_001 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_8.8.0_reindex_temp 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.28 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.23 0 p STARTED ELASTICSEARCH-NODE-1 .ds-ilm-history-5-2023.03.29-000022 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-21 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-15 0 p STARTED ELASTICSEARCH-NODE-1 .ds-ilm-history-5-2023.04.28-000024 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-es-7-2023.05.25 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-22 0 p STARTED ELASTICSEARCH-NODE-1 .transform-internal-007 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-26 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-23 0 p STARTED ELASTICSEARCH-NODE-1 .monitoring-kibana-7-2023.05.24 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-17 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-09 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_security_solution_8.8.0_reindex_temp 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_analytics_8.8.0_001 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-04-28 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_8.6.2_001 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-11 0 p STARTED ELASTICSEARCH-NODE-1 cf_http-2023-05-20 0 p STARTED ELASTICSEARCH-NODE-1 .kibana_alerting_cases_8.8.0_001 0 p STARTED ELASTICSEARCH-NODE-1 ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 28, 2023, 2:31pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/13 "2023-05-28T14:31:39Z")
Hi @A.Hani Back to the previous question what happened to cause this? Looks like your server or disk crashed? First can you run `GET .kibana/_alias` Take a look at the [allocation explain api](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) any time you need insight into allocation issues. Then run allocation explain on one of the regular indices ```auto GET _cluster/allocation/explain?pretty { "index": "cf_http-2023-05-06", "shard": 0, "primary": true } ``` Then run the same on the actual index the the `.kibana` alias is pointing to. Also when you look in the elasticsearch logs do see errors like ```auto Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /var/lib/elasticsearch/nodes/0/indices/3mUO113ES22qeEDguuH-VA/0/_state/state-23.st at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:159) ~[elasticsearch-7.16.3.jar:7.16.3] ``` Most likely the actual elasticsearch data is corrupt from some severe issue, you will probably need to recover from a snapshot unless there is some unusual allocation setting (doubtful) We my might be able to fix the `.kibana` index issue... But I suspect the normal indices going to be bigger problem. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 25, 2023, 2:32pm UTC](https://discuss.elastic.co/t/help-restoring-recreating-security-7/334499/14 "2023-06-25T14:32:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.