Readiness check failure when enabling OIDC authentication

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I'm trying to enable OIDC authentication in elastic/kibana deployed by the operator, having enabled the platinum subscription trial. The operator is starting the required number of pods and they are coming up as running but not ready.

Checking the pod description to find what the readiness check is defined as and exec'ing into the container, I can see that elasticsearch is running and responds to curl on port 9200, however the PROBE_USERNAME and password in the PROBE_PASSWORD_FILE is failing to authenticate:

sh-4.2# curl -vk -u elastic-internal-probe:hmmnr8rchqfp8dm9w4fs87cq https://127.0.0.1:9200
* About to connect() to 127.0.0.1 port 9200 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=edp-mgt-elasticsearch-es-http.monitoring.es.local,OU=edp-mgt-elasticsearch
* 	start date: Sep 19 20:31:52 2019 GMT
* 	expire date: Sep 18 20:41:52 2020 GMT
* 	common name: edp-mgt-elasticsearch-es-http.monitoring.es.local
* 	issuer: CN=edp-mgt-elasticsearch-http,OU=edp-mgt-elasticsearch
* Server auth using Basic with user 'elastic-internal-probe'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYy1pbnRlcm5hbC1wcm9iZTpobW1ucjhyY2hxZnA4ZG05dzRmczg3Y3E=
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:9200
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< content-type: application/json; charset=UTF-8
< content-length: 495
<
* Connection #0 to host 127.0.0.1 left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-internal-probe] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-internal-probe] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

Does anyone have any suggestions on what I can check next ?

2

Hi,

Looks similar to this issue: https://github.com/elastic/cloud-on-k8s/issues/1629
ECK is using the file realm for some internal users, I guess that by adding OIDC authentication you have disabled the file realm.
This should be fixed in the next release. As a workaround you can use some similar settings as the ones described in https://github.com/elastic/cloud-on-k8s/issues/1629#issuecomment-524743103 . This will ensure that the file realm is also enabled.

3

Good spot @michael.morello. I explicitly added the file and native realms and the elasticsearch containers are now coming up as 'ready'. Unfortunately I'm now failing with the Kibana config.

As per https://www.elastic.co/guide/en/elastic-stack-overview/master/oidc-kibana.html I have the following in my kibana CRD:

  config:
    server:
      basePath: /kibana
      rewriteBasePath: true
    xpack.monitoring.enabled: true
    xpack.security.authc.providers: [  oidc, basic ]
    xpack.security.authc.oidc.realm: "ad"
    server.xsrf.whitelist: [ /api/security/v1/oidc ]

I have the trial license activated and see the log output from elasticsearch confirming the trial license is valid, but the kibana contain is going into a CrashLoopBackoff with the following error:

{"type":"log","@timestamp":"2019-09-22T14:47:41Z","tags":["fatal","root"],"pid":1,"message":"{ ValidationError: child \"xpack\" fails because [child \"security\" fails because [child \"authc\" fails because [\"oidc\" is not allowed, \"providers\" is not allowed]]]\n    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)\n    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n    at Config._commit (/usr/share/kibana/src/legacy/server/config/config.js:132:34)\n    at Config.set (/usr/share/kibana/src/legacy/server/config/config.js:102:10)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:74:10)\n    at extendConfigService (/usr/share/kibana/src/legacy/plugin_discovery/plugin_config/extend_config_service.js:45:10) name: 'ValidationError' }"}

 FATAL  ValidationError: child "xpack" fails because [child "security" fails because [child "authc" fails because ["oidc" is not allowed, "providers" is not allowed]]]

When I first saw this error, I assumed that is was because I was only using the Basic license that didnt allow the use of OIDC, but I do now have the trial license active, so guess I'm missing some other config somewhere. I would really appreciate any more pointers please ?

Thanks
Rob