After X-pack installation lost the indexes from kibana

Elastic Stack Kibana
1

Hi,
Recently I have installed the X-Pack in the Kibana AWA in Elastic Search.
Following is my Platform,
OS - CentOS 7.4
Kibana - 6.2
Logstash - 6.2
ES - 6.2
X-Pack - 6.2

Previously before x-pack installation all the beats where working fine and the indexes were created. After xpack installation,the following are the steps done,

Installed the xpack in kibana and elasticsearch
setup-passwords intecrative (created pw for kibana,elastic,logstash)
edited the kibana.yml,filebeat.yml,metricbeat.yml,packetbeat.yml and audit.yml and added the elastic user credentials
restarted all the services
Logs are fine from all the beats

But the indexes that are already created is not showing anything.

Anything I missed?
Thanks for advance
Vishnu

2

A few more clue !
I am getting the following error from the ES logs,

[2018-06-07T13:08:34,809][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:35,290][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:35,564][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:35,805][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:37,625][INFO ][o.e.l.LicenseService ] [node-1] license [bf9b5039-84dc-4455-abd6-375c69695fcc] mode [trial] - valid
[2018-06-07T13:08:37,644][INFO ][o.e.g.GatewayService ] [node-1] recovered [12] indices into cluster_state
[2018-06-07T13:08:37,947][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [elastic]

Please find the below OP too,

curl -u elastic 'http://10.10.114.175:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'elastic':
{
"username" : "elastic",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}

elastic is my ES user. But when I cat my /etc/passwd I get the user as elasticsearch.
I created the elastic user by default x-password, set-password script.
Vishnu

3

I looks like you might have made a mistake during this step:

edited the kibana.yml,filebeat.yml,metricbeat.yml,packetbeat.yml and audit.yml and added the elastic user credentials

one of those services is trying to authenticate with an incorrect password.

4

Hi Tim,
Thanks for the reply. I have double checked all the passwords and configuration files before raising the ticket :frowning: . Please find my beats config. files as follows,
Filebeat OP:-

output.elasticsearch:
Array of hosts to connect to.
hosts: ["http://10.10.114.175:9200"]
username: "elastic"
password: "elastic@1234"

Metricbeat OP:

output.elasticsearch:
hosts: ["10.10.114.175:9200"]
username: "elastic"
password: "elastic@1234"

PakcetBeat OP:

output.elasticsearch:
hosts: ["10.10.114.175:9200"]
username: "elastic"
password: "elastic@1234"

And my Kibana Conf,

elasticsearch.url: "http://10.10.114.175:9200"
elasticsearch.username: "elastic"
elasticsearch.password: "elastic@1234"

All my passwords are correct. Really stuck. Two times I reinstalled from the begening. :frowning:

I have doubt with the following log, from my elasticsearch logs,

[2018-06-07T13:08:37,947][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [elastic]

I have created the password using x-pack 'setup-password interactive' command.

Thanks
Vishnu

5

A few more details,

curl -u elastic -XGET "http://10.10.114.175:9200/_cat/indices"

Enter host password for user 'elastic':
yellow open metricbeat-6.2.4-2018.06.05 MvgkzMJfSru-ANAvYJYC4A 1 1 4156 0 946.9kb 946.9kb
green open .monitoring-es-6-2018.06.09 KMOJ5kbGQ-K7n23d5k46eQ 1 0 257231 738 144.1mb 144.1mb
green open .watcher-history-7-2018.06.10 HM3tjFeaQ0i1da2CjZorng 1 0 11506 0 12.4mb 12.4mb
yellow open metricbeat-6.2.4-2018.06.07 PxA1ZkUnTQ2GeTuNis-WlQ 1 1 74667 0 13.7mb 13.7mb
yellow open packetbeat-6.2.4-2018.06.10 yrPfe_guRXerj7NCLfOq4w 3 1 284508 0 56.6mb 56.6mb
green open .monitoring-es-6-2018.06.07 lopur456S5-mgbcRtIQWhQ 1 0 107359 396 54.5mb 54.5mb
yellow open metricbeat-6.2.4-2018.06.08 pVBTO6_pTYi4D4R-FoNWcw 1 1 130171 0 22.4mb 22.4mb
green open .monitoring-kibana-6-2018.06.09 ZjK2tcbWR8e98NRtVaozYw 1 0 8636 0 1.9mb 1.9mb
yellow open packetbeat-6.2.4-2018.06.09 XRNL7g1gTqiKkQ3zHOCvnQ 3 1 288213 0 56.8mb 56.8mb
yellow open metricbeat-6.2.4-2018.06.10 PlI3B3QdTxOapV4_rj7LFA 1 1 134760 0 22.8mb 22.8mb
green open .monitoring-kibana-6-2018.06.11 HsaV0BjoQpqLvvr4cRhkjA 1 0 1739 0 439.8kb 439.8kb
yellow open filebeat-6.2.4-2018.06.09 AnK2dDXFQPShKRLNsRuk8Q 3 1 11356 0 2.1mb 2.1mb
green open .watcher-history-7-2018.06.07 Bgh5Kx4dQEWBY105eCBsWQ 1 0 7877 0 8.9mb 8.9mb
yellow open packetbeat-6.2.4-2018.06.08 vkfTb9grRKSq_eLfN5L3PA 3 1 321977 0 67mb 67mb
yellow open filebeat-6.2.4-2018.06.07 CtFzilrHRemgYai5Vtl1fQ 3 1 26532 0 4.7mb 4.7mb
green open .monitoring-es-6-2018.06.08 _yTS2wgBQjidHNRbz0o4qQ 1 0 205167 840 115.5mb 115.5mb
green open .monitoring-kibana-6-2018.06.08 J0MOpmNiQumqapgxzy1TLA 1 0 8636 0 1.9mb 1.9mb
yellow open metricbeat-6.2.4-2018.06.11 j3vKzJydTEOgoZQouP7YlQ 1 1 58768 0 10mb 10mb
green open .watcher-history-7-2018.06.09 WgmwkXAFSpajtxM0KfsvMw 1 0 11506 0 12.4mb 12.4mb
yellow open packetbeat-6.2.4-2018.06.07 FW-fcyS4QVq2NjjpiF2YFQ 3 1 151421 0 30.7mb 30.7mb
yellow open filebeat-6.2.4-2018.06.10 m7lsUxg2Q5iH19j8SJkGZg 3 1 11416 0 2.1mb 2.1mb
yellow open filebeat-6.2.4-2018.06.08 DE0z0GymTDaTLoK4gx256g 3 1 24136 0 4.9mb 4.9mb
yellow open auditbeat-6.2.4-2018.06.06 6800YFo9QNGjjE4gSAbG3A 3 1 46 0 40.4kb 40.4kb
green open .monitoring-alerts-6 9m1jz1ZmTumYb6OjIWnOAw 1 0 5 1 36.4kb 36.4kb
green open .monitoring-kibana-6-2018.06.07 FUDPKat4R1atjkv-nttvVg 1 0 5882 0 1.5mb 1.5mb
green open .watcher-history-7-2018.06.08 SZwkUec_T3ydouC6twA_sQ 1 0 11507 0 12.6mb 12.6mb
yellow open filebeat-6.2.4-2018.06.11 P1u3OxY7TT-fsUnsDzHqoQ 3 1 4965 0 1mb 1mb
green open .monitoring-kibana-6-2018.06.10 f6SaPuF0S6iUZ-D4D8Cuhw 1 0 8636 0 1.9mb 1.9mb
green open .watcher-history-7-2018.06.11 xi-G2c7VRVuVMFRmnwYwvw 1 0 2314 0 2.7mb 2.7mb
green open .triggered_watches MOm8ALiESHyPAW-FD449Jw 1 0 0 0 2.2mb 2.2mb
green open .watches mwQWToM0TvquKO7Ubtxu9Q 1 0 6 0 75.7kb 75.7kb
green open .security-6 atRc7CYqQL-SWVkf77dO5A 1 0 3 0 9.9kb 9.9kb
green open .monitoring-es-6-2018.06.10 JzzqmCY_SVmyord5QdifaQ 1 0 309035 936 169.2mb 169.2mb
yellow open metricbeat-6.2.4-2018.06.09 3QOzEimRRO-wVTRBoOMYTQ 1 1 129866 0 22.1mb 22.1mb
yellow open packetbeat-6.2.4-2018.06.11 RQ5LDvHJQd-Dc2SgN4bjFA 3 1 120529 0 24mb 24mb
green open .monitoring-es-6-2018.06.11 k28GHGAMS4C9kKX7JNm1rg 1 0 71883 294 40.7mb 40.7mb
green open .kibana lNg2roUhSKmFLyk9yM5RxA 1 0 375 37 538.6kb 538.6kb

I think elasticsearch is having all my indices. But its not able to load into the kibana!
:frowning:

Vishnu

6

This is also the error from ES logs

[2018-06-11T11:15:10,762][ERROR][o.e.x.s.a.e.ReservedRealm] [qPtXO7b] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.security][doc][reserved-user-elastic]: routing [null]]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:103) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:188) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:183) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:177) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:165) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:190) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:166) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:184) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:217) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:228) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:143) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:113) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:165) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$2(SecurityActionFilter.java:117) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:107) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:117) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:115) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:497) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:73) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.lambda$getReservedUserInfo$16(NativeUsersStore.java:504) ~[?:?]
at org.elasticsearch.xpack.security.support.IndexLifecycleManager.prepareIndexIfNeededThenExecute(IndexLifecycleManager.java:356) ~[?:?]
at org.elasticsearch.xpack.security.SecurityLifecycleService.prepareIndexIfNeededThenExecute(SecurityLifecycleService.java:221) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:503) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:203) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:99) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.doAuthenticateAndCache(CachingUsernamePasswordRealm.java:161) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:100) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:85) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:274) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:310) ~[?:?]

7

This is my Cluster conf,

curl -u elastic http://10.10.114.175:9200/_cluster/health?pretty
Enter host password for user 'elastic':
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 13,
"active_shards" : 13,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 72.22222222222221
}

8

I am facing the same problem. Any assistance would be greatly appreciated as this is a serious bug.

9

@hokiegeek2 Please start a new thread with a description of the problem you're facing.