Received Token Response from OP with status [UNAUTHORIZED]

Elastic Stack Elasticsearch
1

Hi,
I was trying to integrate elasticsearch and kibana with our Okta. However, I had no luck to make it work. I enabled trace and got some error messages as shown below.

It mentioned something like Received Token Response from OP with status [UNAUTHORIZED]. However, the same client_id and client_secret worked for my dummy testing webapp. I was able to auth with my Okta and show all claims it provided.

{"type": "server", "timestamp": "2020-06-10T07:53:41,395Z", "level": "TRACE", "component": "o.e.x.s.a.o.OpenIdConnectAuthenticator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "OpenID Connect Provider redirected user to [/api/security/oidc/callback?code=6Em8ZFoB94HSDX2QQvJQ&state=Jipql2a3mbxZ4WvlGc64_eZU4zpRnZCj6mVkQfaVYfs]. Expected Nonce is [xShLAN69BI7lkcbA3abeB9qshpZA41Fo5wXL64tETn4] and expected State is [Jipql2a3mbxZ4WvlGc64_eZU4zpRnZCj6mVkQfaVYfs]", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "node.id": "sOUiiMS2SpaqMXuZgXNrrQ"  }
{"type": "server", "timestamp": "2020-06-10T07:53:41,899Z", "level": "WARN", "component": "o.e.x.s.a.o.OpenIdConnectAuthenticator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Received Token Response from OP with status [UNAUTHORIZED] and content [{\"error\":\"invalid_client\",\"error_description\":\"The client secret supplied for a confidential client is invalid.\"}]", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "node.id": "sOUiiMS2SpaqMXuZgXNrrQ"  }
{"type": "server", "timestamp": "2020-06-10T07:53:41,900Z", "level": "DEBUG", "component": "o.e.x.s.a.o.OpenIdConnectRealm", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Failed to consume the OpenIdConnectToken ", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "node.id": "sOUiiMS2SpaqMXuZgXNrrQ" ,
"stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: Failed to exchange code for Id Token",
"at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.handleTokenResponse(OpenIdConnectAuthenticator.java:525) [x-pack-security-7.7.1.jar:7.7.1]",
"at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.access$600(OpenIdConnectAuthenticator.java:122) [x-pack-security-7.7.1.jar:7.7.1]",
"at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator$2.completed(OpenIdConnectAuthenticator.java:477) [x-pack-security-7.7.1.jar:7.7.1]",
"at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator$2.completed(OpenIdConnectAuthenticator.java:474) [x-pack-security-7.7.1.jar:7.7.1]",
"at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) [httpcore-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) [httpasyncclient-4.1.4.jar:4.1.4]",
"at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.4.jar:4.1.4]",
"at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.4.jar:4.1.4]",
"at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) [httpcore-nio-4.4.12.jar:4.4.12]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }
{"type": "server", "timestamp": "2020-06-10T07:53:41,906Z", "level": "WARN", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Authentication to realm elk-oidc failed - Failed to authenticate user with OpenID Connect (Caused by ElasticsearchSecurityException[Failed to exchange code for Id Token])", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "node.id": "sOUiiMS2SpaqMXuZgXNrrQ"  }

Instructions and configurations I used:

First, I configured my ELK OIDC web app as shown below.

elk-oidc-app
elk-oidc-app774×868 38.5 KB

Second, I ran the following commands to create k8s secret for keystore, which is basically a file with one line client_secret from my OIDC app.

$ kubectl create secret generic elk-client-secret --from-file=xpack.security.authc.realms.oidc.elk-oidc.rp.client_secret=./elk_client_secret

Then, ran helm command to install elasticsearch and kibana.

$ helm upgrade --install elasticsearch elastic/elasticsearch -f elasticsearch-values.yaml
$ helm upgrade --install kibana elastic/kibana -f kibana-values.yaml

Finally, testing okta authentication flow via port forwarding to kibana service in my k8s cluster.

$ kubectl port-forward svc/kibana-kibana 5601:5601

Settings for elasticsearch-values.yaml and kibana-values.yaml are shown below.

elasticsearch-values.yaml

---
clusterName: "elasticsearch"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.license.self_generated.type: trial
    xpack.security.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.authc.token.enabled: true
    xpack.security.authc.realms.oidc.elk-oidc:
      order: 2
      rp.client_id: "my_client_id"
      rp.response_type: code
      rp.requested_scopes: [openid, profile, email, groups]
      rp.redirect_uri: "https://0.0.0.0:5601/api/security/oidc/callback"
      op.issuer: "https://mydomain.okta.com/oauth2/default"
      op.authorization_endpoint: "https://mydomain.okta.com/oauth2/default/v1/authorize"
      op.token_endpoint: "https://mydomain.okta.com/oauth2/default/v1/token"
      op.jwkset_path: "https://mydomain.okta.com/oauth2/default/v1/keys"
      op.endsession_endpoint: "https://mydomain.okta.com/oauth2/default/v1/logout"
      rp.post_logout_redirect_uri: "https://0.0.0.0:5601/logged_out"
      claims.principal: preferred_username
      claims.groups: "groups"

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username


secretMounts:
 - name: elastic-certificates
   secretName: elastic-certificates
   path: /usr/share/elasticsearch/config/certs

keystore:
  - secretName: elk-client-secret

kibana-values.yaml

---
elasticsearchHosts: "https://elasticsearch-master:9200"

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: 'KIBANA_ENCRYPTION_KEY'
    valueFrom:
      secretKeyRef:
        name: kibana
        key: encryptionkey

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
    xpack.security.authProviders: [oidc, basic]
    xpack.security.authc.oidc.realm: "elk-oidc"
    server.xsrf.whitelist: [/api/security/oidc/callback]

protocol: https

secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs

Is there by any chance anyone could tell what is missing from my configuration? Thanks in advance.

2

Hi @mllu,

I'm almost certain the problem is about how you're passing in the rp.client_secret value. See the relevant part from Configure Elasticsearch for OpenID Connect authentication | Elasticsearch Guide [master] | Elastic (at the end when it talks about the client secret being a secure setting). The value must sit in a special file, the Elasticsearch keystore.
I can't really grok the kubectl create secret generic elk-client-secret cmd so I can be wrong. I can look into how Kubernetes is supposed to work with the elasticsearch keystore if it's still unclear.

3

Hi @Albert_Zaharovits,
yea, I was also thinking that part, but what I did is to just put my client_secret into a file called "elk_client_secret", do I need to base64 encode or do I need to add newline after it?

In my elasticsearch configuration I did reference that like

keystore:
  - secretName: elk-client-secret

I will try to exec into the pods and see if I can get secret then.

4

Digging further, I've found this section about the keystore in the official elasticsearch helm chart. I believe you can make that work in your case, i.e.

kubectl create secret generic elk-oidc-client-secret --from-file=xpack.security.authc.realms.oidc.elk-oidc.rp.client_secret=./elk_client_secret

and add the secret to the elasticsearch keystore:

keystore:
  - secretName: elk-oidc-client-secret

This presumes you're using the official elasticsearch chart. You need some boilerplate code to otherwise make the Kubernetes secrets infra work with elasticsearch secure settings. Elasticsearch secure settings must reside in a particular binary encrypted format to which only the packaged elasticsearch-keytool can write to.

5

thanks for the info, will review those resources. On the other hand, curious if that client is required in elk-oidc-client-secret ?

6

Glad to help. No the "client" part is not required, it's just a name.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.