Upgrade from 6.2 to 6.4 gives "Authentication of [elastic] was terminated by realm [reserved]

Elastic Stack Elasticsearch
1

Hello, after upgrading to elastic 6.4 I am getting the following INFO and ERROR messages when elasticsearch starts.

[2018-10-17T14:51:08,064][INFO ][o.e.x.s.a.AuthenticationService] [_qmypXi] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-10-17T14:51:08,326][ERROR][o.e.x.s.a.e.ReservedRealm] [_qmypXi] failed to retrieve password hash for reserved user [elastic]

I followed the guide https://www.elastic.co/products/upgrade_guide to upgrade, but probably I did something wrong (?).

The system starts fine, and I can logging via Kibana Dashboards.

Are these messages important?
How can I fix the system so they do not appear?

cheers
Yannis

2

Hi Yannis,

It looks like elasticsearch can't access the.security index. What is your cluster status ?

Are there any limes immediately after this one ? There should be a cause for this error close to this.

This is strange as it indicates that your kibana reserved user can authenticate correctly with Elasticsearch - which would not be the case if .security index was unavailable.

Can you share some additional information about your setup ?Do you have any other authentication realms are configured ? Are there other errors in the log or indications that something is wrong when you access via Kibana ?

3

Hi Yannis (!!!)

yes the cluster (... single instance ...) is fine

{
  "cluster_name" : "logging-tst",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 119,
  "active_shards" : 119,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

extra log lines after the error

[2018-10-17T16:09:51,207][ERROR][o.e.x.s.a.e.ReservedRealm] [_qmypXi] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.security][doc][reserved-user-elastic]: routing [null]]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:207) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:172) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:205) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:216) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:170) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:131) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$2(SecurityActionFilter.java:104) ~[?:?]
 [..]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-10-17T16:09:51,240][INFO ][o.e.x.s.a.AuthenticationService] [_qmypXi] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-10-17T16:09:51,252][INFO ][o.e.g.GatewayService     ] [_qmypXi] recovered [119] indices into cluster_state
[2018-10-17T16:09:51,242][ERROR][o.e.x.s.a.e.ReservedRealm] [_qmypXi] failed to retrieve password hash for reserved user [elastic]

Yes, an ldap one (along with the native)

When accessing it using elastic account (via Kibana) or an ldap account, the log in process works fine.

Is it possible that we get the error because .security index was re-created (but the elastic/kibana users have the same password/roles)?

thanks for your support
Yannis

4

Καλησπέρα :slight_smile:

This is what troubles me. It seems you can authenticate as user elastic but then you get

[2018-10-17T14:51:08,326][ERROR][o.e.x.s.a.e.ReservedRealm] [_qmypXi] failed to retrieve password hash for reserved user [elastic]

and I cannot figure out how both of these can be true. Could it be that you have users with a uid that matches one of the reserved users ( i.e. elastic or kibana ) in your ldap ? Is your ldap realm first or second in order ?

What happens when you run

curl -u elastic https://<YOUR_IP_OR_FQDN>:9200/_xpack/security/_authenticate

curl -u kibana https://<YOUR_IP_OR_FQDN>:9200/_xpack/security/_authenticate

How did you recreate the .security index ? Did you run setup-passwords after recreating the .security index ?

5

Hello,

the curl commands respond fine

curl -k  -u kibana:XX http://XX:9200/_xpack/security/_authenticate?pretty  
{
  "username" : "kibana",
  "roles" : [
    "kibana_system"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true
}


curl -k -u elastic:XX http://XX:9200/_xpack/security/_authenticate?pretty
{
  "username" : "elastic",
  "roles" : [
    "superuser"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true
}

Yes (if I remember correctly..., I deleted accidentally all indices, including .security and then searched online how can I restore x-pack accounts, I recreated them with the same passwords )

LDAP is second with order : 2, No kibana or elastic users exist in our LDAP, moreover the issue was there even when only native realm was present.

Another (maybe related) finding I have is the error shown bellow

elastic
elastic1159×600 19.2 KB

cheers
Yannis

6

Reading through your previous message, @Albert_Zaharovits spotted that It looks like something attempts to authenticate as elastic 12 ms before .security index is recovered

  • Do you have anything else connecting to Elasticsearch as elastic user?
  • Do you get these [2018-10-17T16:09:51,240][INFO ][o.e.x.s.a.AuthenticationService] [_qmypXi] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic] errors only on startup ?
7

we have a few beats directly connecting to elasticsearch (temporary using the elastic account...).
Most probably these are the suspects causing the ERROR message!

thanks for the support!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.