# Elastisearch doesn't create .security index after loss of data

**URL:** <https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [June 2, 2023, 10:27pm UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123 "2023-06-02T22:27:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Savva\_Morozov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/savva_morozov/32/116837_2.png) [@Savva\_Morozov](https://discuss.elastic.co/u/Savva_Morozov)\
**Post date:** [June 2, 2023, 10:27pm UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123/1 "2023-06-02T22:27:10Z")

</div>

Hello! We are using Elasticsearch and Kibana on Kubernetes deployed via Helm charts and recently we occasionaly deleted all the data from persistent volumes that our two nodes Elasticsearch cluster uses. Since then we cannot enter into Kibana which says "Kibana server is not ready yet." and in the Kibana log it shows error `[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security]`. It made sense after deletion of all indices from the volumes but the problem is that even after we reinstalled the Elasticsearch it still doesn't create .security index even though it shows in the log that `Security is enabled`. API call `GET /_cat/indices?expand_wildcards=all` shows just this `green open .geoip_databases FdIHiGqgQyCHa66t9BL34g 1 1 42 0 79.9mb 39.9mb`. What can be the reason that prevents Elasticsearch from creating .security index?

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [June 3, 2023, 8:52am UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123/2 "2023-06-03T08:52:42Z")

</div>

The security index is only created when there is something to write to it.

It's perfectly normal for a brand new cluster to have no security index until you access one of the security APIs (e.g. to create a user, reset a password, etc).

---

<div class="post-metadata">

**Author:** ![Savva\_Morozov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/savva_morozov/32/116837_2.png) [@Savva\_Morozov](https://discuss.elastic.co/u/Savva_Morozov)\
**Post date:** [June 3, 2023, 12:39pm UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123/3 "2023-06-03T12:39:09Z")

</div>

Thank you for the answer. But in this case does the Kibana's request suppose to create .security index?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 3, 2023, 1:20pm UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123/4 "2023-06-03T13:20:50Z")

</div>

> [@Savva\_Morozov](#):
>
> But in this case does the Kibana's request suppose to create .security index?

I don't think so, If I'm not wrong kibana will first try to read the index, since it does not exists it will give you this error.

> [@Savva\_Morozov](#):
>
> but the problem is that even after we reinstalled the Elasticsearch it still doesn't create .security index

When you reinstalled Elasticsearch did you run the steps to create the internal users again? How did you reinstalled it?

---

<div class="post-metadata">

**Author:** ![Savva\_Morozov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/savva_morozov/32/116837_2.png) [@Savva\_Morozov](https://discuss.elastic.co/u/Savva_Morozov)\
**Post date:** [June 4, 2023, 2:30pm UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123/5 "2023-06-04T14:30:40Z")

</div>

> [@leandrojmp](#):
>
> I don't think so, If I'm not wrong kibana will first try to read the index, since it does not exists it will give you this error.

So at what moment does the Elasticsearch create .security index then?

> [@leandrojmp](#):
>
> When you reinstalled Elasticsearch did you run the steps to create the internal users again? How did you reinstalled it?

I reinstalled it deleting the Elasticsearch Helm chart and installing this chart again. I didn't run any addtitional steps to create the internal users.

But the problem seems to be solved after I also reinstalled the Kibana Helm chart. It didn't work smoothly and I had to clean some things manually but in the end it worked. Looks like the problem was on  
the side of the Kibana not being able to securely communicate with the Elasticsearch because of the problem with the token. It looks so for me because during the deinstallation I saw an error of the post deletion job with the text: `Cleaning the Kibana Elasticsearch token Cleaning token statusCode: 404 {"found":false}`  
As I understood as the original .security index was deleted, the token that the Kibana used wasn't actually valid anymore. The strange part that the was no error logs saying about that.

Is there any documentation on the way Kibana communicates with Elasticsearch using tokens especially when deployed on Kubernetes? I really would like to understand this process better.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 2, 2023, 2:31pm UTC](https://discuss.elastic.co/t/elastisearch-doesnt-create-security-index-after-loss-of-data/335123/6 "2023-07-02T14:31:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
