I have an ES 7.16.2 cluster running with TLS set up. I haven't set the xpack.security.enabled setting in my elasticsearch.yml file. I'm using the BASIC license. So it should have used the default value of false for that setting according to the docs.
I have these other tls settings in the elasticsearch.yml file -
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
I'm trying to upgrade ES to 8.8.2. But before I do that, I was looking at the deprecation logs and saw this log ( security_implicitly_disabled) saying -
The default behavior of disabling security on basic licenses is deprecated. In a later version of Elasticsearch, the value of [xpack.security.enabled] will default to "true" , regardless of the license level. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security, or explicitly disable security by setting [xpack.security.enabled] to false in elasticsearch.yml
so I was trying to either set it to true or false (whatever works) and proceed with the upgrade, but seems like I'm stuck as neither of the options work for me.
If I set xpack.security.enabled to true, then the curl requests start failing with -
$ curl localhost:9200/_cat/nodes?pretty
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
}
},
"status" : 401
}
If I set it to false, then the node doesn't seem to discover the master node -
[2023-07-20 22:25:24.929454] [2023-07-20T22:25:24,928][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:25:34.931068] [2023-07-20T22:25:34,930][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:25:44.932605] [2023-07-20T22:25:44,932][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:25:44.934415] [2023-07-20T22:25:44,934][WARN ][o.e.n.Node ] [node-name] timed out while waiting for initial discovery state - timeout: 30s
[2023-07-20 22:25:44.960853] [2023-07-20T22:25:44,960][INFO ][o.e.h.AbstractHttpServerTransport] [node-name] publish_address {10.100.63.3:9200}, bound_addresses {127.0.0.1:9200}, {10.100.63.3:9200}
[2023-07-20 22:25:44.961255] [2023-07-20T22:25:44,961][INFO ][o.e.n.Node ] [node-name] started
[2023-07-20 22:25:54.933836] [2023-07-20T22:25:54,933][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:26:04.934501] [2023-07-20T22:26:04,934][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:26:14.935798] [2023-07-20T22:26:14,935][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
[2023-07-20 22:26:15.291376] [2023-07-20T22:26:15,283][WARN ][r.suppressed ] [node-name] path: /_cluster/health, params: {}
[2023-07-20 22:26:15.291443] org.elasticsearch.discovery.MasterNotDiscoveredException: null
[2023-07-20 22:26:15.291502] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291534] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291567] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291603] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291647] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.291675] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[2023-07-20 22:26:15.291709] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[2023-07-20 22:26:15.291729] at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-07-20 22:26:15.310346] [2023-07-20T22:26:15,309][WARN ][r.suppressed ] [node-name] path: /_cluster/health, params: {}
[2023-07-20 22:26:15.310388] org.elasticsearch.discovery.MasterNotDiscoveredException: null
[2023-07-20 22:26:15.310414] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310435] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310485] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310534] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310556] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:15.310579] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[2023-07-20 22:26:15.310663] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[2023-07-20 22:26:15.310699] at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-07-20 22:26:16.397061] [2023-07-20T22:26:16,396][WARN ][r.suppressed ] [node-name] path: /_cluster/health, params: {pretty=true}
[2023-07-20 22:26:16.397163] org.elasticsearch.discovery.MasterNotDiscoveredException: null
[2023-07-20 22:26:16.397209] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397246] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397293] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397327] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397359] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
[2023-07-20 22:26:16.397386] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[2023-07-20 22:26:16.397442] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[2023-07-20 22:26:16.397469] at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-07-20 22:26:24.937242] [2023-07-20T22:26:24,936][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-name] master not discovered yet: have discovered [{node-name}{1IHJq5jiTJOEhK-Yel8-_g}{NMQvwsJMTR-OonJhl5sxYQ}{10.100.63.3}{10.100.63.3:9300}{d}]; discovery will continue using [10.100.78.134:9300, 10.100.54.148:9300, 10.100.63.3:9300, 10.100.56.132:9300, 10.100.87.115:9300, 10.100.47.190:9300, 10.100.43.18:9300, 10.100.31.222:9300, 10.100.79.169:9300] from hosts providers and [] from last-known cluster state; node term 136, last-accepted version 609274 in term 136
My questions are -
-
How come the xpack.security implicitly disabled when the TLS is enabled? (I've confirmed the nodes are communicating using TLS because if I remove those tls settings from one of the node and restart it, then it doesn't join the cluster - which is expected as other nodes have it enabled). Doesn't ES need this xpack.security.enabled to be set to true as well to start using TLS?
-
Is it possible to enable xpack.security but not enforce the authentication? I don't want to setup client authentication as there are too many clients and many clusters that I'm trying to upgrade. So it'd be great if I can keep TLS enabled without enforcing authentication. I don't have an option to set xpack.security.enabled to false as doing so probably disables TLS as well and the node with this setting = false can't join the cluster anymore.
-
How come nodes are joining the cluster just fine right now with xpack.security.enabled implicitly disabled, but when I explicitly disable it then they can't seem to be able to join the cluster?
I can't upgrade to ES 8 because that will change the xpack.security.enabled setting to true and the nodes won't be able to join the cluster. I can provide any other useful info if needed. Thanks!