# ElasticsearchSecurityException when security is enabled on master node but not the data nodes

**URL:** https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076
**Category:** Elasticsearch
**Tags:** elastic-stack-security
**Created:** [August 31, 2023, 4:36pm UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076 "2023-08-31T16:36:05Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![darshanypatel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshanypatel/32/123722_2.png) [@darshanypatel](https://discuss.elastic.co/u/darshanypatel)
#### Post date: [August 31, 2023, 4:36pm UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/1 "2023-08-31T16:36:05Z")

</div>

I have an ES 7.16.2 cluster running with dedicated master nodes and separate data nodes.  
If/when -

1. `xpack.security.enabled` is set to `true` on the master nodes
2. some of the data nodes have xpack security disabled
3. anonymous access is enabled on all the nodes with the anonymous role having all access to index / cluster operations -

```auto
anonymous_role:
  cluster: ['all']
  indices:
    - names: ['*']
      privileges: ['all']

```

and

```auto
xpack.security.authc.anonymous.roles: anonymous_role
xpack.security.authc.anonymous.authz_exception: true

```

then I see these errors logs on the data nodes -

```auto
[<timestamp>] [WARN][o.e.c.a.s.ShardStateAction] [<data-node-host-name>] unexpected failure while sending request [internal:cluster/shard/failure] to [{<master-node-host-name}{hash}{hash}{IP}{IP:Port}{m}{zone=z, xpack.installed=true, transform.node=false}] for shard entry [shard id [[xyz][1]], allocation id [hash], primary term [0], message [shard failure, reason [primary shard [[xyz][1], node[hash], [P], s[STARTED], a[id=hash]] was demoted while failing replica shard]], failure [ElasticsearchSecurityException[action [internal:cluster/shard/failure] is unauthorized for user [_anonymous] with roles [anonymous_role]]], markAsStale [true]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: action [internal:cluster/shard/failure] is unauthorized for user [_anonymous] with roles [anonymous_role]
	at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:34) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:928) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:870) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:471) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:371) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:256) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:138) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:173) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:279) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:144) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:127) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:258) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:136) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:128) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:415) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:106) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:88) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:743) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:147) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:119) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:84) [elasticsearch-7.16.2.jar:7.16.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:71) [transport-netty4-client-7.16.2.jar:7.16.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

```

and also these -

```auto
[<timestamp>] [WARN][o.e.i.e.Engine] [<data-node-host-name>] [xyz][1] failed engine [primary shard [[xyz][1], node[hash], [P], s[STARTED], a[id=hash]] was demoted while failing replica shard]
[<timestamp>] org.elasticsearch.ElasticsearchSecurityException: action [internal:cluster/shard/failure] is unauthorized for user [_anonymous] with roles [anonymous_role]

```

On the server nodes, I see these errors -

```auto
[<timestamp>][WARN][o.e.c.r.a.AllocationService] [<master-node-host-name>] failing shard [failed shard, shard [xyz][1], node[hash], [P], s[STARTED], a[id=hash], message [master {<master-node-host-name>}{hash}{hash}{IP}{IP:Port}{m}{zone=z, xpack.installed=true, transform.node=false} has not removed previously failed shard. resending shard failure], failure [null], markAsStale [true]]
[<timestamp>][WARN][o.e.c.r.a.AllocationService] [<master-node-host-name>] [index-name][507] marking unavailable shards as stale: [hash]
[<timestamp>][WARN][o.e.x.s.a.AuthorizationService] [<master-node-host-name>] denying access as action [internal:cluster/shard/failure] is not an index or cluster action

```

I have these questions -

1. Why is the anonymous user is assumed for inter-node communication? shouldn't that be used only for http access?
2. Why is the `all` privilege on indices & cluster not enough for an action? What other privilege would allow that action `[internal:cluster/shard/failure]` that was being denied for the anonymous role? I couldn't find anything related in [the elastic docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-privileges.html).

---

<div class="post-metadata">

### Author: ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)
#### Post date: [September 5, 2023, 9:35pm UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/2 "2023-09-05T21:35:57Z")

</div>

You cannot have security enabled on only some nodes, it has to be enabled on every node in the cluster.

The errors you are seeing are a consequence of trying to run in an unsupported configuration.

> **[Set up minimal security for Elasticsearch | Elasticsearch Guide \[8.9\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-minimal-setup.html#_enable_elasticsearch_security_features)**

---

<div class="post-metadata">

### Author: ![darshanypatel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshanypatel/32/123722_2.png) [@darshanypatel](https://discuss.elastic.co/u/darshanypatel)
#### Post date: [September 8, 2023, 11:25am UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/3 "2023-09-08T11:25:15Z")

</div>

I know we cannot have security enabled on only some nodes, but that will be the case when I am enabling security (for the first time) in a live cluster by restarting nodes one by one?

Is it safe to say that if I enable security on all the data nodes first, and then enable it on master nodes then I won't see these errors? or is it all because of some config that I'm using incorrectly?

> The errors you are seeing are a consequence of trying to run in an unsupported configuration.

Can you please elaborate a bit? What unsupported configuration can cause these ElasticsearchSecurityExceptions?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 8, 2023, 11:30am UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/4 "2023-09-08T11:30:30Z")

</div>

> [@darshanypatel](#):
>
> but that will be the case when I am enabling security (for the first time) in a live cluster by restarting nodes one by one

Enabling security [requires a full cluster restart](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/security-basic-setup.html) (it has to be applied to all nodes at the same time) and can not be done through a rolling upgrade.

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [September 8, 2023, 11:39am UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/5 "2023-09-08T11:39:42Z")

</div>

> [@darshanypatel](#):
>
> Can you please elaborate a bit? What unsupported configuration can cause these ElasticsearchSecurityExceptions?

The unsupported configuration is having some nodes with security enable and some nodes without security enabled, this will not work.

When you enable or disable security you need to do a full cluster restarted, the nodes with a different configuration will stop working until you restart them to use the updated configuration, it doesn't matter the order.

A master node with security enabled will not allow any data node without security enabled to connect, and a data node with security enabled will not connect to a master node without security, so a full cluster restart is required.

---

<div class="post-metadata">

### Author: ![darshanypatel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshanypatel/32/123722_2.png) [@darshanypatel](https://discuss.elastic.co/u/darshanypatel)
#### Post date: [September 21, 2023, 6:28am UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/6 "2023-09-21T06:28:09Z")

</div>

I had missed the point that we need a full cluster restart to enable security.

I have a cluster that has security enabled on all the nodes now (although it was not enabled with a full cluster restart - rather randomly enabled it one by one across all the nodes). But I still got these errors for some bulk requests after the full cluster had security enabled and in place -

> "error":{"root\_cause":[{"type":"security\_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [\_anonymous] with roles [\<anonymous\_role\>], this action is granted by the index privileges [create\_doc,create,delete,index,write,all]"}],"status":403}

Is this also because I had enabled the security without doing a full cluster restart? [All the nodes have `xpack.security.enabled: true` now and anonymous role has access to all the index / cluster operations]

What could have caused these errors now? Also, this doesn't happen on every bulk request. This happened for a very few requests.

---

<div class="post-metadata">

### Author: ![darshanypatel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshanypatel/32/123722_2.png) [@darshanypatel](https://discuss.elastic.co/u/darshanypatel)
#### Post date: [September 22, 2023, 7:44pm UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/7 "2023-09-22T19:44:26Z")

</div>

I checked the logs and found that these errors were due to an internal issue that my configuration management system had caused, so ignore my last response. I'm assuming that everything will work fine as long as all the nodes in the cluster have security enabled and I won't see these errors again. Thank you!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 20, 2023, 7:44pm UTC](https://discuss.elastic.co/t/elasticsearchsecurityexception-when-security-is-enabled-on-master-node-but-not-the-data-nodes/342076/8 "2023-10-20T19:44:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.