# After X-pack installation lost the indexes from kibana

**URL:** <https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921>\
**Category:** Kibana\
**Created:** [June 7, 2018, 7:54am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921 "2018-06-07T07:54:22Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![vishnuduttpv](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\
**Post date:** [June 7, 2018, 7:54am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/1 "2018-06-07T07:54:23Z")

</div>

Hi,  
Recently I have installed the X-Pack in the Kibana AWA in Elastic Search.  
Following is my Platform,  
OS - CentOS 7.4  
Kibana - 6.2  
Logstash - 6.2  
ES - 6.2  
X-Pack - 6.2

Previously before x-pack installation all the beats where working fine and the indexes were created. After xpack installation,the following are the steps done,

> Installed the xpack in kibana and elasticsearch  
> setup-passwords intecrative (created pw for kibana,elastic,logstash)  
> edited the kibana.yml,filebeat.yml,metricbeat.yml,packetbeat.yml and audit.yml and added the elastic user credentials  
> restarted all the services  
> Logs are fine from all the beats

But the indexes that are already created is not showing anything.

Anything I missed?  
Thanks for advance  
Vishnu

---

<div class="post-metadata">

**Author:** ![vishnuduttpv](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\
**Post date:** [June 7, 2018, 8:07am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/2 "2018-06-07T08:07:47Z")

</div>

A few more clue !  
I am getting the following error from the ES logs,

> [2018-06-07T13:08:34,809][INFO][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]  
> [2018-06-07T13:08:35,290][INFO][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]  
> [2018-06-07T13:08:35,564][INFO][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]  
> [2018-06-07T13:08:35,805][INFO][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]  
> [2018-06-07T13:08:37,625][INFO][o.e.l.LicenseService] [node-1] license [bf9b5039-84dc-4455-abd6-375c69695fcc] mode [trial] - valid  
> [2018-06-07T13:08:37,644][INFO][o.e.g.GatewayService] [node-1] recovered [12] indices into cluster\_state  
> [2018-06-07T13:08:37,947][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [elastic]

Please find the below OP too,

> curl -u elastic '[http://10.10.114.175:9200/\_xpack/security/\_authenticate?pretty](http://10.10.114.175:9200/_xpack/security/_authenticate?pretty)'  
> Enter host password for user 'elastic':  
> {  
> "username" : "elastic",  
> "roles" : [  
> "superuser"  
> ],  
> "full\_name" : null,  
> "email" : null,  
> "metadata" : {  
> "\_reserved" : true  
> },  
> "enabled" : true  
> }

elastic is my ES user. But when I cat my /etc/passwd I get the user as elasticsearch.  
I created the elastic user by default x-password, set-password script.  
Vishnu

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [June 8, 2018, 2:51am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/3 "2018-06-08T02:51:09Z")

</div>

I looks like you might have made a mistake during this step:

> edited the kibana.yml,filebeat.yml,metricbeat.yml,packetbeat.yml and audit.yml and added the elastic user credentials

one of those services is trying to authenticate with an incorrect password.

---

<div class="post-metadata">

**Author:** ![vishnuduttpv](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\
**Post date:** [June 11, 2018, 4:29am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/4 "2018-06-11T04:29:25Z")

</div>

Hi Tim,  
Thanks for the reply. I have double checked all the passwords and configuration files before raising the ticket ☹ . Please find my beats config. files as follows,  
Filebeat OP:-

> output.elasticsearch:  
> Array of hosts to connect to.  
> hosts: ["[http://10.10.114.175:9200](http://10.10.114.175:9200)"]  
> username: "elastic"  
> password: "elastic@1234"

Metricbeat OP:

> output.elasticsearch:  
> hosts: ["10.10.114.175:9200"]  
> username: "elastic"  
> password: "elastic@1234"

PakcetBeat OP:

> output.elasticsearch:  
> hosts: ["10.10.114.175:9200"]  
> username: "elastic"  
> password: "elastic@1234"

And my Kibana Conf,

> elasticsearch.url: "[http://10.10.114.175:9200](http://10.10.114.175:9200)"  
> elasticsearch.username: "elastic"  
> elasticsearch.password: "elastic@1234"

All my passwords are correct. Really stuck. Two times I reinstalled from the begening. ☹

I have doubt with the following log, from my elasticsearch logs,

> [2018-06-07T13:08:37,947][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [elastic]

I have created the password using x-pack 'setup-password interactive' command.

Thanks  
Vishnu

---

<div class="post-metadata">

**Author:** ![vishnuduttpv](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\
**Post date:** [June 11, 2018, 4:51am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/5 "2018-06-11T04:51:02Z")

</div>

A few more details,

curl -u elastic -XGET "[http://10.10.114.175:9200/\_cat/indices](http://10.10.114.175:9200/_cat/indices)"

> Enter host password for user 'elastic':  
> yellow open metricbeat-6.2.4-2018.06.05 MvgkzMJfSru-ANAvYJYC4A 1 1 4156 0 946.9kb 946.9kb  
> green open .monitoring-es-6-2018.06.09 KMOJ5kbGQ-K7n23d5k46eQ 1 0 257231 738 144.1mb 144.1mb  
> green open .watcher-history-7-2018.06.10 HM3tjFeaQ0i1da2CjZorng 1 0 11506 0 12.4mb 12.4mb  
> yellow open metricbeat-6.2.4-2018.06.07 PxA1ZkUnTQ2GeTuNis-WlQ 1 1 74667 0 13.7mb 13.7mb  
> yellow open packetbeat-6.2.4-2018.06.10 yrPfe\_guRXerj7NCLfOq4w 3 1 284508 0 56.6mb 56.6mb  
> green open .monitoring-es-6-2018.06.07 lopur456S5-mgbcRtIQWhQ 1 0 107359 396 54.5mb 54.5mb  
> yellow open metricbeat-6.2.4-2018.06.08 pVBTO6\_pTYi4D4R-FoNWcw 1 1 130171 0 22.4mb 22.4mb  
> green open .monitoring-kibana-6-2018.06.09 ZjK2tcbWR8e98NRtVaozYw 1 0 8636 0 1.9mb 1.9mb  
> yellow open packetbeat-6.2.4-2018.06.09 XRNL7g1gTqiKkQ3zHOCvnQ 3 1 288213 0 56.8mb 56.8mb  
> yellow open metricbeat-6.2.4-2018.06.10 PlI3B3QdTxOapV4\_rj7LFA 1 1 134760 0 22.8mb 22.8mb  
> green open .monitoring-kibana-6-2018.06.11 HsaV0BjoQpqLvvr4cRhkjA 1 0 1739 0 439.8kb 439.8kb  
> yellow open filebeat-6.2.4-2018.06.09 AnK2dDXFQPShKRLNsRuk8Q 3 1 11356 0 2.1mb 2.1mb  
> green open .watcher-history-7-2018.06.07 Bgh5Kx4dQEWBY105eCBsWQ 1 0 7877 0 8.9mb 8.9mb  
> yellow open packetbeat-6.2.4-2018.06.08 vkfTb9grRKSq\_eLfN5L3PA 3 1 321977 0 67mb 67mb  
> yellow open filebeat-6.2.4-2018.06.07 CtFzilrHRemgYai5Vtl1fQ 3 1 26532 0 4.7mb 4.7mb  
> green open .monitoring-es-6-2018.06.08 \_yTS2wgBQjidHNRbz0o4qQ 1 0 205167 840 115.5mb 115.5mb  
> green open .monitoring-kibana-6-2018.06.08 J0MOpmNiQumqapgxzy1TLA 1 0 8636 0 1.9mb 1.9mb  
> yellow open metricbeat-6.2.4-2018.06.11 j3vKzJydTEOgoZQouP7YlQ 1 1 58768 0 10mb 10mb  
> green open .watcher-history-7-2018.06.09 WgmwkXAFSpajtxM0KfsvMw 1 0 11506 0 12.4mb 12.4mb  
> yellow open packetbeat-6.2.4-2018.06.07 FW-fcyS4QVq2NjjpiF2YFQ 3 1 151421 0 30.7mb 30.7mb  
> yellow open filebeat-6.2.4-2018.06.10 m7lsUxg2Q5iH19j8SJkGZg 3 1 11416 0 2.1mb 2.1mb  
> yellow open filebeat-6.2.4-2018.06.08 DE0z0GymTDaTLoK4gx256g 3 1 24136 0 4.9mb 4.9mb  
> yellow open auditbeat-6.2.4-2018.06.06 6800YFo9QNGjjE4gSAbG3A 3 1 46 0 40.4kb 40.4kb  
> green open .monitoring-alerts-6 9m1jz1ZmTumYb6OjIWnOAw 1 0 5 1 36.4kb 36.4kb  
> green open .monitoring-kibana-6-2018.06.07 FUDPKat4R1atjkv-nttvVg 1 0 5882 0 1.5mb 1.5mb  
> green open .watcher-history-7-2018.06.08 SZwkUec\_T3ydouC6twA\_sQ 1 0 11507 0 12.6mb 12.6mb  
> yellow open filebeat-6.2.4-2018.06.11 P1u3OxY7TT-fsUnsDzHqoQ 3 1 4965 0 1mb 1mb  
> green open .monitoring-kibana-6-2018.06.10 f6SaPuF0S6iUZ-D4D8Cuhw 1 0 8636 0 1.9mb 1.9mb  
> green open .watcher-history-7-2018.06.11 xi-G2c7VRVuVMFRmnwYwvw 1 0 2314 0 2.7mb 2.7mb  
> green open .triggered\_watches MOm8ALiESHyPAW-FD449Jw 1 0 0 0 2.2mb 2.2mb  
> green open .watches mwQWToM0TvquKO7Ubtxu9Q 1 0 6 0 75.7kb 75.7kb  
> green open .security-6 atRc7CYqQL-SWVkf77dO5A 1 0 3 0 9.9kb 9.9kb  
> green open .monitoring-es-6-2018.06.10 JzzqmCY\_SVmyord5QdifaQ 1 0 309035 936 169.2mb 169.2mb  
> yellow open metricbeat-6.2.4-2018.06.09 3QOzEimRRO-wVTRBoOMYTQ 1 1 129866 0 22.1mb 22.1mb  
> yellow open packetbeat-6.2.4-2018.06.11 RQ5LDvHJQd-Dc2SgN4bjFA 3 1 120529 0 24mb 24mb  
> green open .monitoring-es-6-2018.06.11 k28GHGAMS4C9kKX7JNm1rg 1 0 71883 294 40.7mb 40.7mb  
> green open .kibana lNg2roUhSKmFLyk9yM5RxA 1 0 375 37 538.6kb 538.6kb

I think elasticsearch is having all my indices. But its not able to load into the kibana!  
☹

Vishnu

---

<div class="post-metadata">

**Author:** ![vishnuduttpv](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\
**Post date:** [June 11, 2018, 5:54am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/6 "2018-06-11T05:54:52Z")

</div>

This is also the error from ES logs

> [2018-06-11T11:15:10,762][ERROR][o.e.x.s.a.e.ReservedRealm] [qPtXO7b] failed to retrieve password hash for reserved user [elastic]  
> org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.security][doc][reserved-user-elastic]: routing [null]]  
> at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:103) ~[?:?]  
> at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:188) ~[?:?]  
> at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:183) ~[?:?]  
> at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:177) ~[?:?]  
> at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:165) ~[?:?]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:190) ~[?:?]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:166) ~[?:?]  
> at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:184) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:217) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:228) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:182) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:143) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:113) ~[?:?]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:165) ~[?:?]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$2(SecurityActionFilter.java:117) ~[?:?]  
> at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:107) ~[?:?]  
> at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:117) ~[?:?]  
> at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:115) ~[?:?]  
> at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:497) ~[elasticsearch-6.2.4.jar:6.2.4]  
> at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:73) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.lambda$getReservedUserInfo$16(NativeUsersStore.java:504) ~[?:?]  
> at org.elasticsearch.xpack.security.support.IndexLifecycleManager.prepareIndexIfNeededThenExecute(IndexLifecycleManager.java:356) ~[?:?]  
> at org.elasticsearch.xpack.security.SecurityLifecycleService.prepareIndexIfNeededThenExecute(SecurityLifecycleService.java:221) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:503) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:203) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:99) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.doAuthenticateAndCache(CachingUsernamePasswordRealm.java:161) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:100) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:85) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:274) ~[?:?]  
> at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:93) ~[?:?]  
> at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:310) ~[?:?]

---

<div class="post-metadata">

**Author:** ![vishnuduttpv](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\
**Post date:** [June 11, 2018, 6:00am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/7 "2018-06-11T06:00:19Z")

</div>

This is my Cluster conf,

curl -u elastic [http://10.10.114.175:9200/\_cluster/health?pretty](http://10.10.114.175:9200/_cluster/health?pretty)  
Enter host password for user 'elastic':  
{  
"cluster\_name" : "elasticsearch",  
"status" : "yellow",  
"timed\_out" : false,  
"number\_of\_nodes" : 1,  
"number\_of\_data\_nodes" : 1,  
"active\_primary\_shards" : 13,  
"active\_shards" : 13,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 5,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 72.22222222222221  
}

---

<div class="post-metadata">

**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\
**Post date:** [July 6, 2018, 12:01pm UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/8 "2018-07-06T12:01:46Z")

</div>

I am facing the same problem. Any assistance would be greatly appreciated as this is a serious bug.

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [July 9, 2018, 5:00am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/9 "2018-07-09T05:00:55Z")

</div>

@hokiegeek2 Please start a new thread with a description of the problem you're facing.

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [July 9, 2018, 5:01am UTC](https://discuss.elastic.co/t/after-x-pack-installation-lost-the-indexes-from-kibana/134921/10 "2018-07-09T05:01:00Z")

</div>