Enable xpack.security but not password authentication

Elastic Stack Elasticsearch
1

I have an ES 7.16.2 cluster running with TLS set up. I haven't set the xpack.security.enabled setting in my elasticsearch.yml file. I'm using the BASIC license. So it should have used the default value of false for that setting according to the docs.

I have these other tls settings in the elasticsearch.yml file -

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

I'm trying to upgrade ES to 8.8.2. But before I do that, I was looking at the deprecation logs and saw this log ( security_implicitly_disabled) saying -

The default behavior of disabling security on basic licenses is deprecated. In a later version of Elasticsearch, the value of [xpack.security.enabled] will default to "true" , regardless of the license level. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security, or explicitly disable security by setting [xpack.security.enabled] to false in elasticsearch.yml

so I was trying to either set it to true or false (whatever works) and proceed with the upgrade, but seems like I'm stuck as neither of the options work for me.

If I set xpack.security.enabled to true, then the curl requests start failing with -

$ curl localhost:9200/_cat/nodes?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

If I set it to false, then the node doesn't seem to discover the master node -

[2023-07-20 22:25:24.929454] [2023-07-20T22:25:24,928][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:25:34.931068] [2023-07-20T22:25:34,930][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:25:44.932605] [2023-07-20T22:25:44,932][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:25:44.934415] [2023-07-20T22:25:44,934][WARN ][o.e.n.Node               ] [node-name] timed out while waiting for initial discovery state - timeout: 30s
[2023-07-20 22:25:44.960853] [2023-07-20T22:25:44,960][INFO ][o.e.h.AbstractHttpServerTransport] [node-name] publish_address {10.100.63.3:9200}, bound_addresses {127.0.0.1:9200}, {10.100.63.3:9200}
[2023-07-20 22:25:44.961255] [2023-07-20T22:25:44,961][INFO ][o.e.n.Node               ] [node-name] started
[2023-07-20 22:25:54.933836] [2023-07-20T22:25:54,933][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:26:04.934501] [2023-07-20T22:26:04,934][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:26:14.935798] [2023-07-20T22:26:14,935][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:26:15.291376] [2023-07-20T22:26:15,283][WARN ][r.suppressed             ] [node-name] path: /_cluster/health, params: {}
[2023-07-20 22:26:15.291443] org.elasticsearch.discovery.MasterNotDiscoveredException: null
[2023-07-20 22:26:15.291502] 	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291534] 	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291567] 	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291603] 	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291647] 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291675] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[2023-07-20 22:26:15.291709] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[2023-07-20 22:26:15.291729] 	at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-07-20 22:26:15.310346] [2023-07-20T22:26:15,309][WARN ][r.suppressed             ] [node-name] path: /_cluster/health, params: {}
[2023-07-20 22:26:15.310388] org.elasticsearch.discovery.MasterNotDiscoveredException: null
[2023-07-20 22:26:15.310414] 	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310435] 	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310485] 	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310534] 	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310556] 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310579] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[2023-07-20 22:26:15.310663] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[2023-07-20 22:26:15.310699] 	at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-07-20 22:26:16.397061] [2023-07-20T22:26:16,396][WARN ][r.suppressed             ] [node-name] path: /_cluster/health, params: {pretty=true}
[2023-07-20 22:26:16.397163] org.elasticsearch.discovery.MasterNotDiscoveredException: null
[2023-07-20 22:26:16.397209] 	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397246] 	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397293] 	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397327] 	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397359] 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397386] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[2023-07-20 22:26:16.397442] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[2023-07-20 22:26:16.397469] 	at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-07-20 22:26:24.937242] [2023-07-20T22:26:24,936][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136

My questions are -

  1. How come the xpack.security implicitly disabled when the TLS is enabled? (I've confirmed the nodes are communicating using TLS because if I remove those tls settings from one of the node and restart it, then it doesn't join the cluster - which is expected as other nodes have it enabled). Doesn't ES need this xpack.security.enabled to be set to true as well to start using TLS?

  2. Is it possible to enable xpack.security but not enforce the authentication? I don't want to setup client authentication as there are too many clients and many clusters that I'm trying to upgrade. So it'd be great if I can keep TLS enabled without enforcing authentication. I don't have an option to set xpack.security.enabled to false as doing so probably disables TLS as well and the node with this setting = false can't join the cluster anymore.

  3. How come nodes are joining the cluster just fine right now with xpack.security.enabled implicitly disabled, but when I explicitly disable it then they can't seem to be able to join the cluster?

I can't upgrade to ES 8 because that will change the xpack.security.enabled setting to true and the nodes won't be able to join the cluster. I can provide any other useful info if needed. Thanks!

2

This is a complicated side effect of the fact that licensing for security changed in Elasticsearch 7.1 combined with the fact the license for a cluster is stored in cluster state, and dynamically changeable.

The consequence of those circumstances means that when you start a node in ES 7.x, it doesn't know what license it has, or whether security should default to on or off.
So, while we documented that security is "off" by default in ES 7.x on a basic license, that's a simplification - by default it's in a half-on state, where it supports TLS (so it can join other nodes in the cluster) and then disables the rest of security (authentication, RBAC) when it determines that the cluster has a basic license. And security can be automatically re-activated if you install a paid license into the cluster.

That behaviour does not exist in ES 8.x

Is it possible to enable xpack.security but not enforce the authentication?

Yes. You can enable security, and then turn on anonymous access. That will have roughly the same effect.
You can decide what role your anonymous users have. For example, if you have a lot of search clients, you could make the anonymous only have read access and then configure any ingest/management clients with a username/password.

As described above, implicitly disabled isn't actually the same as explicitly disabled. The default behaviour for basic licensed nodes is to have security enabled, but inactive.

3

Thank you for the clarification!

I've used the anonymous access as you've mentioned and it works well with security enabled without asking for password. So I'm unblocked now.

Good to know some of the details you've given that is not there in the elasticsearch documentation.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.