After X-pack installation lost the indexes from kibana

vishnuduttpv · June 7, 2018, 7:54am

Hi,
Recently I have installed the X-Pack in the Kibana AWA in Elastic Search.
Following is my Platform,
OS - CentOS 7.4
Kibana - 6.2
Logstash - 6.2
ES - 6.2
X-Pack - 6.2

Previously before x-pack installation all the beats where working fine and the indexes were created. After xpack installation,the following are the steps done,

Installed the xpack in kibana and elasticsearch
setup-passwords intecrative (created pw for kibana,elastic,logstash)
edited the kibana.yml,filebeat.yml,metricbeat.yml,packetbeat.yml and audit.yml and added the elastic user credentials
restarted all the services
Logs are fine from all the beats

But the indexes that are already created is not showing anything.

Anything I missed?
Thanks for advance
Vishnu

vishnuduttpv · June 7, 2018, 8:07am

A few more clue !
I am getting the following error from the ES logs,

[2018-06-07T13:08:34,809][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:35,290][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:35,564][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:35,805][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-06-07T13:08:37,625][INFO ][o.e.l.LicenseService ] [node-1] license [bf9b5039-84dc-4455-abd6-375c69695fcc] mode [trial] - valid
[2018-06-07T13:08:37,644][INFO ][o.e.g.GatewayService ] [node-1] recovered [12] indices into cluster_state
[2018-06-07T13:08:37,947][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [elastic]

Please find the below OP too,

curl -u elastic 'http://10.10.114.175:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'elastic':
{
"username" : "elastic",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}

elastic is my ES user. But when I cat my /etc/passwd I get the user as elasticsearch.
I created the elastic user by default x-password, set-password script.
Vishnu

TimV · June 8, 2018, 2:51am

I looks like you might have made a mistake during this step:

edited the kibana.yml,filebeat.yml,metricbeat.yml,packetbeat.yml and audit.yml and added the elastic user credentials

one of those services is trying to authenticate with an incorrect password.

vishnuduttpv · June 11, 2018, 4:29am

Hi Tim,
Thanks for the reply. I have double checked all the passwords and configuration files before raising the ticket . Please find my beats config. files as follows,
Filebeat OP:-

output.elasticsearch:
Array of hosts to connect to.
hosts: ["http://10.10.114.175:9200"]
username: "elastic"
password: "elastic@1234"

Metricbeat OP:

output.elasticsearch:
hosts: ["10.10.114.175:9200"]
username: "elastic"
password: "elastic@1234"

PakcetBeat OP:

output.elasticsearch:
hosts: ["10.10.114.175:9200"]
username: "elastic"
password: "elastic@1234"

And my Kibana Conf,

elasticsearch.url: "http://10.10.114.175:9200"
elasticsearch.username: "elastic"
elasticsearch.password: "elastic@1234"

All my passwords are correct. Really stuck. Two times I reinstalled from the begening.

I have doubt with the following log, from my elasticsearch logs,

[2018-06-07T13:08:37,947][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [elastic]

I have created the password using x-pack 'setup-password interactive' command.

Thanks
Vishnu

vishnuduttpv · June 11, 2018, 4:51am

A few more details,

curl -u elastic -XGET "http://10.10.114.175:9200/_cat/indices"

Enter host password for user 'elastic': yellow open metricbeat-6.2.4-2018.06.05 green open .monitoring-es-6-2018.06.09 green open .watcher-history-7-2018.06.10 yellow open metricbeat-6.2.4-2018.06.07 yellow open packetbeat-6.2.4-2018.06.10 green open .monitoring-es-6-2018.06.07 yellow open metricbeat-6.2.4-2018.06.08 green open .monitoring-kibana-6-2018.06.09 yellow open packetbeat-6.2.4-2018.06.09 yellow open metricbeat-6.2.4-2018.06.10 green open .monitoring-kibana-6-2018.06.11 yellow open filebeat-6.2.4-2018.06.09 green open .watcher-history-7-2018.06.07 yellow open packetbeat-6.2.4-2018.06.08 yellow open filebeat-6.2.4-2018.06.07 green open .monitoring-es-6-2018.06.08 green open .monitoring-kibana-6-2018.06.08 yellow open metricbeat-6.2.4-2018.06.11 green open .watcher-history-7-2018.06.09 yellow open packetbeat-6.2.4-2018.06.07 yellow open filebeat-6.2.4-2018.06.10 yellow open filebeat-6.2.4-2018.06.08 yellow open auditbeat-6.2.4-2018.06.06 green open .monitoring-alerts-6 green open .monitoring-kibana-6-2018.06.07 green open .watcher-history-7-2018.06.08 yellow open filebeat-6.2.4-2018.06.11 green open .monitoring-kibana-6-2018.06.10 green open .watcher-history-7-2018.06.11 green open .triggered_watches green open .watches green open .security-6 green open .monitoring-es-6-2018.06.10 yellow open metricbeat-6.2.4-2018.06.09 yellow open packetbeat-6.2.4-2018.06.11 green open .monitoring-es-6-2018.06.11 green open .kibana
MvgkzMJfSru-ANAvYJYC4A 1 1 4156 0 946.9kb 946.9kb
KMOJ5kbGQ-K7n23d5k46eQ 1 0 257231 738 144.1mb 144.1mb
HM3tjFeaQ0i1da2CjZorng 1 0 11506 0 12.4mb 12.4mb
PxA1ZkUnTQ2GeTuNis-WlQ 1 1 74667 0 13.7mb 13.7mb
yrPfe_guRXerj7NCLfOq4w 3 1 284508 0 56.6mb 56.6mb
lopur456S5-mgbcRtIQWhQ 1 0 107359 396 54.5mb 54.5mb
pVBTO6_pTYi4D4R-FoNWcw 1 1 130171 0 22.4mb 22.4mb
ZjK2tcbWR8e98NRtVaozYw 1 0 8636 0 1.9mb 1.9mb
XRNL7g1gTqiKkQ3zHOCvnQ 3 1 288213 0 56.8mb 56.8mb
PlI3B3QdTxOapV4_rj7LFA 1 1 134760 0 22.8mb 22.8mb
HsaV0BjoQpqLvvr4cRhkjA 1 0 1739 0 439.8kb 439.8kb
AnK2dDXFQPShKRLNsRuk8Q 3 1 11356 0 2.1mb 2.1mb
Bgh5Kx4dQEWBY105eCBsWQ 1 0 7877 0 8.9mb 8.9mb
vkfTb9grRKSq_eLfN5L3PA 3 1 321977 0 67mb 67mb
CtFzilrHRemgYai5Vtl1fQ 3 1 26532 0 4.7mb 4.7mb
_yTS2wgBQjidHNRbz0o4qQ 1 0 205167 840 115.5mb 115.5mb
J0MOpmNiQumqapgxzy1TLA 1 0 8636 0 1.9mb 1.9mb
j3vKzJydTEOgoZQouP7YlQ 1 1 58768 0 10mb 10mb
WgmwkXAFSpajtxM0KfsvMw 1 0 11506 0 12.4mb 12.4mb
FW-fcyS4QVq2NjjpiF2YFQ 3 1 151421 0 30.7mb 30.7mb
m7lsUxg2Q5iH19j8SJkGZg 3 1 11416 0 2.1mb 2.1mb
DE0z0GymTDaTLoK4gx256g 3 1 24136 0 4.9mb 4.9mb
6800YFo9QNGjjE4gSAbG3A 3 1 46 0 40.4kb 40.4kb
9m1jz1ZmTumYb6OjIWnOAw 1 0 5 1 36.4kb 36.4kb
FUDPKat4R1atjkv-nttvVg 1 0 5882 0 1.5mb 1.5mb
SZwkUec_T3ydouC6twA_sQ 1 0 11507 0 12.6mb 12.6mb
P1u3OxY7TT-fsUnsDzHqoQ 3 1 4965 0 1mb 1mb
f6SaPuF0S6iUZ-D4D8Cuhw 1 0 8636 0 1.9mb 1.9mb
xi-G2c7VRVuVMFRmnwYwvw 1 0 2314 0 2.7mb 2.7mb
MOm8ALiESHyPAW-FD449Jw 1 0 0 0 2.2mb 2.2mb
mwQWToM0TvquKO7Ubtxu9Q 1 0 6 0 75.7kb 75.7kb
atRc7CYqQL-SWVkf77dO5A 1 0 3 0 9.9kb 9.9kb
JzzqmCY_SVmyord5QdifaQ 1 0 309035 936 169.2mb 169.2mb
3QOzEimRRO-wVTRBoOMYTQ 1 1 129866 0 22.1mb 22.1mb
RQ5LDvHJQd-Dc2SgN4bjFA 3 1 120529 0 24mb 24mb
k28GHGAMS4C9kKX7JNm1rg 1 0 71883 294 40.7mb 40.7mb
lNg2roUhSKmFLyk9yM5RxA 1 0 375 37 538.6kb 538.6kb

I think elasticsearch is having all my indices. But its not able to load into the kibana!

Vishnu

vishnuduttpv · June 11, 2018, 5:54am

This is also the error from ES logs

[2018-06-11T11:15:10,762][ERROR][o.e.x.s.a.e.ReservedRealm] [qPtXO7b] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.security][doc][reserved-user-elastic]: routing [null]]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:103) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:188) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:183) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:177) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:165) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:190) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:166) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:184) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:217) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:228) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:143) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:113) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:165) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$2(SecurityActionFilter.java:117) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:107) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:117) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:115) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:497) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:73) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.lambda$getReservedUserInfo$16(NativeUsersStore.java:504) ~[?:?]
at org.elasticsearch.xpack.security.support.IndexLifecycleManager.prepareIndexIfNeededThenExecute(IndexLifecycleManager.java:356) ~[?:?]
at org.elasticsearch.xpack.security.SecurityLifecycleService.prepareIndexIfNeededThenExecute(SecurityLifecycleService.java:221) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:503) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:203) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:99) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.doAuthenticateAndCache(CachingUsernamePasswordRealm.java:161) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:100) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:85) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:274) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:310) ~[?:?]

vishnuduttpv · June 11, 2018, 6:00am

This is my Cluster conf,

curl -u elastic http://10.10.114.175:9200/_cluster/health?pretty
Enter host password for user 'elastic':
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 13,
"active_shards" : 13,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 72.22222222222221
}

hokiegeek2 · July 6, 2018, 12:01pm

I am facing the same problem. Any assistance would be greatly appreciated as this is a serious bug.

TimV · July 9, 2018, 5:00am

@hokiegeek2 Please start a new thread with a description of the problem you're facing.