Received Token Response from OP with status [UNAUTHORIZED]

I was trying to integrate elasticsearch and kibana with our Okta. However, I had no luck to make it work. I enabled trace and got some error messages as shown below.

It mentioned something like Received Token Response from OP with status [UNAUTHORIZED]. However, the same client_id and client_secret worked for my dummy testing webapp. I was able to auth with my Okta and show all claims it provided.

{"type": "server", "timestamp": "2020-06-10T07:53:41,395Z", "level": "TRACE", "component": "o.e.x.s.a.o.OpenIdConnectAuthenticator", "": "elasticsearch", "": "elasticsearch-master-0", "message": "OpenID Connect Provider redirected user to [/api/security/oidc/callback?code=6Em8ZFoB94HSDX2QQvJQ&state=Jipql2a3mbxZ4WvlGc64_eZU4zpRnZCj6mVkQfaVYfs]. Expected Nonce is [xShLAN69BI7lkcbA3abeB9qshpZA41Fo5wXL64tETn4] and expected State is [Jipql2a3mbxZ4WvlGc64_eZU4zpRnZCj6mVkQfaVYfs]", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "": "sOUiiMS2SpaqMXuZgXNrrQ"  }
{"type": "server", "timestamp": "2020-06-10T07:53:41,899Z", "level": "WARN", "component": "o.e.x.s.a.o.OpenIdConnectAuthenticator", "": "elasticsearch", "": "elasticsearch-master-0", "message": "Received Token Response from OP with status [UNAUTHORIZED] and content [{\"error\":\"invalid_client\",\"error_description\":\"The client secret supplied for a confidential client is invalid.\"}]", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "": "sOUiiMS2SpaqMXuZgXNrrQ"  }
{"type": "server", "timestamp": "2020-06-10T07:53:41,900Z", "level": "DEBUG", "component": "o.e.x.s.a.o.OpenIdConnectRealm", "": "elasticsearch", "": "elasticsearch-master-0", "message": "Failed to consume the OpenIdConnectToken ", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "": "sOUiiMS2SpaqMXuZgXNrrQ" ,
"stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: Failed to exchange code for Id Token",
"at [x-pack-security-7.7.1.jar:7.7.1]",
"at$600( [x-pack-security-7.7.1.jar:7.7.1]",
"at$2.completed( [x-pack-security-7.7.1.jar:7.7.1]",
"at$2.completed( [x-pack-security-7.7.1.jar:7.7.1]",
"at org.apache.http.concurrent.BasicFuture.completed( [httpcore-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted( [httpasyncclient-4.1.4.jar:4.1.4]",
"at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady( [httpasyncclient-4.1.4.jar:4.1.4]",
"at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady( [httpasyncclient-4.1.4.jar:4.1.4]",
"at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.BaseIOReactor.readable( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.BaseIOReactor.execute( [httpcore-nio-4.4.12.jar:4.4.12]",
"at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$ [httpcore-nio-4.4.12.jar:4.4.12]",
"at [?:?]"] }
{"type": "server", "timestamp": "2020-06-10T07:53:41,906Z", "level": "WARN", "component": "o.e.x.s.a.AuthenticationService", "": "elasticsearch", "": "elasticsearch-master-0", "message": "Authentication to realm elk-oidc failed - Failed to authenticate user with OpenID Connect (Caused by ElasticsearchSecurityException[Failed to exchange code for Id Token])", "cluster.uuid": "54GI7fVTRT2xwbH-kQ0yAQ", "": "sOUiiMS2SpaqMXuZgXNrrQ"  }

Instructions and configurations I used:

First, I configured my ELK OIDC web app as shown below.

Second, I ran the following commands to create k8s secret for keystore, which is basically a file with one line client_secret from my OIDC app.

$ kubectl create secret generic elk-client-secret

Then, ran helm command to install elasticsearch and kibana.

$ helm upgrade --install elasticsearch elastic/elasticsearch -f elasticsearch-values.yaml
$ helm upgrade --install kibana elastic/kibana -f kibana-values.yaml

Finally, testing okta authentication flow via port forwarding to kibana service in my k8s cluster.

$ kubectl port-forward svc/kibana-kibana 5601:5601

Settings for elasticsearch-values.yaml and kibana-values.yaml are shown below.


clusterName: "elasticsearch"
nodeGroup: "master"

  master: "true"
  ingest: "true"
  data: "true"

protocol: https

  elasticsearch.yml: |
    xpack.license.self_generated.type: trial true true /usr/share/elasticsearch/config/certs/elastic-certificates.p12 /usr/share/elasticsearch/config/certs/elastic-certificates.p12 true certificate /usr/share/elasticsearch/config/certs/elastic-certificates.p12 /usr/share/elasticsearch/config/certs/elastic-certificates.p12 true
      order: 2
      rp.client_id: "my_client_id"
      rp.response_type: code
      rp.requested_scopes: [openid, profile, email, groups]
      rp.redirect_uri: ""
      op.issuer: ""
      op.authorization_endpoint: ""
      op.token_endpoint: ""
      op.jwkset_path: ""
      op.endsession_endpoint: ""
      rp.post_logout_redirect_uri: ""
      claims.principal: preferred_username
      claims.groups: "groups"

        name: elastic-credentials
        key: password
        name: elastic-credentials
        key: username

 - name: elastic-certificates
   secretName: elastic-certificates
   path: /usr/share/elasticsearch/config/certs

  - secretName: elk-client-secret


elasticsearchHosts: "https://elasticsearch-master:9200"

        name: elastic-credentials
        key: username
        name: elastic-credentials
        key: password
        name: kibana
        key: encryptionkey

  kibana.yml: |
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem ${KIBANA_ENCRYPTION_KEY}
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate [oidc, basic] "elk-oidc"
    server.xsrf.whitelist: [/api/security/oidc/callback]

protocol: https

  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs

Is there by any chance anyone could tell what is missing from my configuration? Thanks in advance.

Hi @mllu,

I'm almost certain the problem is about how you're passing in the rp.client_secret value. See the relevant part from Configure Elasticsearch for OpenID Connect authentication | Elasticsearch Guide [master] | Elastic (at the end when it talks about the client secret being a secure setting). The value must sit in a special file, the Elasticsearch keystore.
I can't really grok the kubectl create secret generic elk-client-secret cmd so I can be wrong. I can look into how Kubernetes is supposed to work with the Elasticsearch keystore if it's still unclear.

Hi @Albert_Zaharovits,
yea, I was also thinking that part, but what I did is to just put my client_secret into a file called "elk_client_secret", do I need to base64 encode or do I need to add newline after it?

In my elasticsearch configuration I did reference that like

  - secretName: elk-client-secret

I will try to exec into the pods and see if I can get secret then.

Digging further, I've found this section about the keystore in the official elasticsearch helm chart. I believe you can make that work in your case, i.e.

kubectl create secret generic elk-oidc-client-secret

and add the secret to the elasticsearch keystore:

  - secretName: elk-oidc-client-secret

This presumes you're using the official elasticsearch chart. You need some boilerplate code to otherwise make the Kubernetes secrets infra work with elasticsearch secure settings. Elasticsearch secure settings must reside in a particular binary encrypted format to which only the packaged elasticsearch-keytool can write to.

thanks for the info, will review those resources. On the other hand, curious if that client is required in elk-oidc-client-secret ?

Glad to help. No the "client" part is not required, it's just a name.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.