Can't find keystore data?

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I'm trying to get OpenID working with ECK, but its not finding the client_secret key/value in the keystore.

I have verified that it (seems to be) populating the keystore. If I set up elasticsearch without the OpenID realm:

apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elasticsearch-poc
spec:
  version: 7.2.0
  secureSettings:
    secretName: azure-openid-secret
  nodes:
    - nodeCount: 1
      config:
        node.master: true
        node.data: true
        node.ingest: true
        xpack.security.authc.token.enabled: true
      volumeClaimTemplates:
      - metadata:
          name: elasticdata
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
          storageClassName: default

You can see the keystore gets the appropriate data:

[root@elasticsearch-poc-es-5kvf75d48q elasticsearch]# cd /mnt/elastic/secure-settings/
[root@elasticsearch-poc-es-5kvf75d48q secure-settings]# ls -al
total 4
drwxrwxrwt 3 root root  100 Jul 24 20:18 .
drwxr-xr-x 9 root root 4096 Jul 24 20:20 ..
drwxr-xr-x 2 root root   60 Jul 24 20:18 ..2019_07_24_20_18_31.193342258
lrwxrwxrwx 1 root root   31 Jul 24 20:18 ..data -> ..2019_07_24_20_18_31.193342258
lrwxrwxrwx 1 root root   62 Jul 24 20:18 xpack.security.authc.realms.oidc.oidc1.rp.client_secret -> ..data/xpack.security.authc.realms.oidc.oidc1.rp.client_secret

[root@elasticsearch-poc-es-5kvf75d48q config]# elasticsearch-keystore list
keystore.seed
xpack.security.authc.realms.oidc.oidc1.rp.client_secret

(I looked in the xpack.security.authc.realms.oidc.oidc1.rp.client_secret file, and it has the correct data.)

Now, if I add the OpenID stuff into nodes config (just under the xpack.security.authc.token.enabled line):

        xpack.security.authc.realms.oidc.oidc1:
          order: 10
          rp.client_id: "REDACTED"
          rp.response_type: "code"
          rp.redirect_uri: "REDACTED"
          op.issuer: "REDACTED"
          op.authorization_endpoint: "REDACTED"
          op.token_endpoint: "REDACTED"
          op.userinfo_endpoint: "REDACTED"
          op.endsession_endpoint: "REDACTED"
          op.jwkset_path: "REDACTED"
          rp.post_logout_redirect_uri: "REDACTED"
          claims.principal: sub

And kubectl apply the changes, the new pod spins up, only to give me a stacktrace, the relevant line being:

"Caused by: org.elasticsearch.common.settings.SettingsException: The configuration setting [xpack.security.authc.realms.oidc.oidc1.rp.client_secret] is required",

So, what am I missing that it can't find what is obviously there?

2

@JoeyLemur Are you still running into this? Can you add what version of the operator you're running if so? My tests below are in 0.9:

I was trying to repro your issue but I had mixed success. I did end up getting that error to happen once when I was toying around with the config, but then tried to re-create the error from scratch and could not. I did see that in that case, the keystore init container didn't get added to the pod. That should be happening if the operator does not think there are any secure settings to add, but unfortunately I didn't dive deeper before deleting the instance and trying again.

I was able to get the pod to start successfully with a slightly modified version of the example you provided:

---
apiVersion: v1
kind: Secret
metadata:
  name: azure-openid-secret
type: Opaque
data:
  xpack.security.authc.realms.oidc.oidc1.rp.client_secret: YWRtaW4=
...
---
apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elasticsearch-poc
spec:
  version: 7.2.0
  secureSettings:
    secretName: azure-openid-secret
  nodes:
    - nodeCount: 1
      config:
        node.master: true
        node.data: true
        node.ingest: true
        xpack.security.authc.token.enabled: true
        xpack.security.authc.realms.oidc.oidc1:
          order: 10
          rp.client_id: "clientid"
          rp.response_type: "code"
          rp.redirect_uri: "redirecturi"
          op.issuer: "https://accounts.google.com"
          op.authorization_endpoint: "https://accounts.google.com/o/oauth2/v2/auth"
          op.token_endpoint: "https://oauth2.googleapis.com/token"
          op.userinfo_endpoint: "https://openidconnect.googleapis.com/v1/userinfo"
          op.endsession_endpoint: ""
          op.jwkset_path: "https://www.googleapis.com/oauth2/v3/certs"
          rp.post_logout_redirect_uri: "logout_uri"
          claims.principal: sub
      volumeClaimTemplates:
      - metadata:
          name: elasticdata
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

Though obviously not with a usable configuration, it did get further than the error you mentioned. I also tried creating it with the xpack.security.authc.realms.oidc.oidc1 section commented out, then re-applying a config with it to follow the steps you mentioned but still couldn't reproduce it.

3

Yeah, this seems to have been fixed in 0.9 (along with a number of other things I was having problems with): I get as far as it complaining that I don't have the right license for doing OpenID now. :slight_smile:

4

Glad to hear it, thanks for following up