6.3.2 to 6.5.0 - can't get cluster nodes to come up with SSL - Google Cloud

[liquid_es]

Trying spin up a cluster in either 6.4.0 or 6.5.0 with a configuration that works on 6.3.2:

• 1. Anonymous user mapped to superuser role
• 2. http and transport traffic SSL encrypted

Relevant parts of elasticsearch.yml below.

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.http.ssl.key: my.key
xpack.security.http.ssl.certificate: my.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/my.ca.pem" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: my.key
xpack.security.transport.ssl.certificate: my.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/my.ca.pem" ]
xpack.ssl.verification_mode: certificate

xpack.security.authc:
  anonymous:
    roles: superuser

network.host: _gce:hostname_,_local_
http.host: 0.0.0.0
plugin.mandatory: discovery-gce
cloud:
  gce:
    project_id: my_project
    zone: my_zone
discovery:
  zen.hosts_provider: gce

Error on master when nodes try to join from transport trace:
_xpack_security_authenticationhs6LxAgAKX2Fub255bW91cwEJc3VwZXJ1c2VyCgEJX3Jlc2VydmVkBQEAAAEAB01Yd1dlM0ULX19hbm9ueW1vdXMLX19hbm9ueW1vdXMA...g.:[MXwWe3E][10.178.0.20:9300][internal:discovery/zen/fd/ping]....Mactionl:discovery/zen/fd/ping]thorizedr8org.elasticsearch.xpack.core.security.support.Exceptions..Exceptions.java.authorizationError.;org.elasticsearch.xpack.security.authz.AuthorizationService..AuthorizationService.java.denialException..;org.elasticsearch.xpack.security.authz.AuthorizationService..AuthorizationService.java.denial..;org.elasticsearch.xpack.security.authz.AuthorizationService..AuthorizationService.java.authorize..Lorg.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile..ServerTransportFilter.java.lambda$inbound$2..Iorg.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer..AuthorizationUtils.java.maybeRun..Iorg.elastic

Seems like my role mapping is being ignored. Anonymous should not be denied access.

[Yogesh_Gaikwad]

Hi @liquid_es,

Could you please format the error stack trace? so we can see what's happening.

Thanks and Regards,
Yogesh Gaikwad

[liquid_es]

Hi Yogesh,

The transport trace logs are full of garbage and hard to format. The forum editor is also give me trouble. Is this better? It's saying anonymous is not authorized to perform "discovery/zen/fd/ping".

xpack_security_authenticationhs6LxAgAKX2Fub255bW91cwEJc3VwZXJ1c2VyCgEJX3Jlc2VydmVkBQEAAAEAB01Yd1dlM0ULX19hbm9ueW1vdXMLX19hbm9ueW1vdXMA
:[MXwWe3E][10.178.0.20:9300][internal:discovery/zen/fd/ping]....action:discovery/zen/fd/ping]unauthorized
org.elasticsearch.xpack.core.security.support.Exceptions
Exceptions.java.authorizationError.
org.elasticsearch.xpack.security.authz.AuthorizationService
AuthorizationService.java.denialException
org.elasticsearch.xpack.security.authz.AuthorizationService
AuthorizationService.java.denial
org.elasticsearch.xpack.security.authz.AuthorizationService
AuthorizationService.java.authorize
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile..ServerTransportFilter.java.lambda$inbound$2
org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer..AuthorizationUtils.java.maybeRun..Iorg.elastic

[liquid_es]

I am running with no special license. I noticed in 6.4.0 and later that I could not use SSL for http and transport without:

xpack.security.enabled: true

I don't want any security features except SSL encryption. These errors during node discovery happen when I enable security. Am I out of luck to want SSL encryption without the other security features?

I can't even list the built in roles without a license:

[root@logs-sandbox-a-fh0c ~]# curl -s --insecure https://localhost:9200/_xpack/security/role
{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}

[root@logs-sandbox-a-fh0c ~]# curl -s --insecure https://localhost:9200/_xpack/security/role/superuser
{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}

[Christian_Dahlqvist]

As outlined on the subscriptions page, Security (which includes SSL) requires a commercial license.

[liquid_es]

That was NOT listed as a breaking change in the 6.4.0 release notes and it's a whopper.

Not very nice to rip that out of the free version after 6.3.2 and then make it extraordinarily difficult to troubleshoot rather than throwing an obvious error at startup.

Adding the following tokens so others hopefully can benefit.

Elasticsearch 6.4.0 6.5.0 6.5.1 SSL disabled broken license

[Christian_Dahlqvist]

As far as I know that was the case in version 6.3 as well. Let's see if someone else can clarify.

[liquid_es]

Not here. Same config as above on another cluster. No license.

[root@my-cluster-b-r931 ~]# curl -s --insecure https://localhost:9200
{
  "name" : "c30Qxve",
  "cluster_name" : "my_cluster",
  "cluster_uuid" : "Dg3bWZc8TQ6JqVkvTL1qTw",
  "version" : {
    "number" : "6.3.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "053779d",
    "build_date" : "2018-07-20T05:20:23.451332Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

[root@my-cluster-b-r931 ~]# openssl s_client -connect localhost:9300
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

[TimV (Tim Vernum)]

SSL was never officially included in the free license, but under certain circumstances it would work.
This was related to the order in which nodes join a cluster relative to when the license is checked (the license is stored in the cluster, so it's not possible to do those checks at node boot).

In recent releases we changed some of the implementation details, and some configurations that might have previously worked, no longer do. This wasn't done intentionally, it was just a consequence of other changes, but because this was never a supported configuration it wasn't identified as a breaking change.