During a rolling upgrade running into denying access as action [internal:cluster/coordination/join/validate] is not an index or cluster action. I'm trying to figure out what is causing this and if any way forward. I have a few different things going on with this upgrade, will outline all of that and steps taken here.
I'm attempting the following upgrade path: Elasticsearch 7.8.0 OSS -> Elasticsearch 7.17.21 -> Elasticsearch 8.3.2
Details:
Elasticsearch 7.8.0 OSS (no xpack)
Elasticsearch 7.17.21 (have tried with/without xpack here)
Elasticsearch 8.3.2 (xpack)
Relevant Elasticsearch settings.
Elasticsearch 7.17.21
xpack.security.enabled: true
xpack.security.fips_mode.enabled: false # can't be enabled with a basic license
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.client_authentication: none
xpack.security.transport.ssl.keystore.path: /pathtokeystore
xpack.security.transport.ssl.truststore.path: /pathtotruststore
xpack.security.transport.ssl.keystore.type: JKS
xpack.security.transport.ssl.truststore.type: JKS
xpack.security.transport.ssl.supported_protocols: TLSv1.3,TLSv1.2
xpack.security.http.ssl.enabled: false
xpack.security.authc.anonymous.username: anonymous_user
xpack.security.authc.anonymous.roles: superuser
xpack.security.authc.anonymous.authz_exception: true
Elasticsearch 8.3.2
xpack.security.enabled: true
xpack.security.autoconfiguration.enabled: false
xpack.security.fips_mode.enabled: false # can't be enabled with a basic license
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /pathtokeystore
xpack.security.transport.ssl.truststore.path: /pathtotruststore
xpack.security.transport.ssl.keystore.type: JKS
xpack.security.transport.ssl.truststore.type: JKS
xpack.security.transport.ssl.supported_protocols: TLSv1.3,TLSv1.2
xpack.security.http.ssl.enabled: false
xpack.security.authc.anonymous.username: anonymous_user
xpack.security.authc.anonymous.roles: superuser
xpack.security.authc.anonymous.authz_exception: true
I have got working 7.8.0 to 7.17.21 (without xpack), I can do this rolling upgrade. When attempting to roll into 8.3.2 I hit denying access as action. I tried 7.8.0 to 7.17.21 (with xpack) I see the same issue.
When looking at the cluster existing nodes see the new ones but new nodes can't join old cluster (theory here is addition of xpack is problem). I've also deployed 7.17.21 -> 8.3.2 rolling upgrade and this works, only difference I see is the starting point with xpack.
Stacktrace
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [clustername][ip:9300][internal:cluster/coordination/join]",
"Caused by: java.lang.IllegalStateException: failure when sending a validation request to node",
"at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:512) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1173) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
"at java.lang.Thread.run(Thread.java:829) [?:?]",
"Caused by: org.elasticsearch.transport.RemoteTransportException: [loadbalancer-worker-es-example-service-10-25-213-3][10.25.213.3:9300][internal:cluster/coordination/join/validate]",
"Caused by: org.elasticsearch.ElasticsearchSecurityException: action [internal:cluster/coordination/join/validate] is unauthorized for user [anonymous_user] with roles [superuser]",
"at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:34) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:925) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:867) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:468) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:377) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:262) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:148) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:277) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:154) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:137) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:264) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:136) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:377) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.handleNullToken(AuthenticatorChain.java:319) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:132) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.21.jar:7.17.21]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:132) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:166) ~[?:?]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:166) ~[?:?]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:166) ~[?:?]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:166) ~[?:?]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]",
I looked at implementing an authorization engine plugin Authorization Engine Plugin Example but this won't work since internal actions are never routed to it.
Is it possible to add xpack during a rolling upgrade on a cluster that previously didn't have it?
Any help here would be greatly appreciated.