Kibana service account not working for v8 upgrade deprecation note

[rsaeks]

We recently moved our stack to 7.17 and are working through some of the warnings in the 8.0 upgrade assistant to be ready for the new version in the future. When changing Kibana to use Elasticsearch.serviceAccountToken instead of the Elasticsearch.username and Elasticsearch.password options.

Following the link, we then used the CLI tool Elasticsearch-service-tokens on one of our Elasticsearch nodes to generate the elastic/kibana account.

When commenting out the username / password options and entering in the serviceAccountToken setting instead, we are unable to start Kibana with the following error message:

"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [kibana-token]"

It is a unclear from the documentation if the service account needs to be created on each Elasticsearch node with the CLI tool if one node is fine or if the service_accounts file needs to be copied once created.

Could anyone provide some clarification?

Thank you

[jportner (Joe Portner)]

@rsaeks

It is a unclear from the documentation if the service account needs to be created on each Elasticsearch node with the CLI tool if one node is fine or if the service_accounts file needs to be copied once created.

You only need to generate the service account token on one node in your cluster. It behaves similarly to a native realm user in that sense. EDIT: This is inaccurate, see Yang's reply below.

I have a couple questions:

1. Are you able to use curl to authenticate to Elasticsearch directly with the service account token?
   • If you can't authenticate directly with curl, then your token is just invalid.
2. Do you have a separate monitoring cluster configured (using monitoring.ui.elasticsearch.hosts)?
   • If so, you'll need to generate a separate service account token on your monitoring cluster and configure that with monitoring.ui.elasticsearch.serviceAccountToken. It looks like that option is missing from our Monitoring docs, I added a new issue for it here.

[rsaeks]

[quote="jportner, post:2, topic:296125"]

• Are you able to use curl to authenticate to Elasticsearch directly with the service account token?

When I use the curl command this is the output that comes back: {"username":"elastic/kibana","roles":[],"full_name":"Service account - elastic/kibana","email":null,"token":{"name":"kibana-token","type":"_service_account_file"},"metadata":{"_elastic_service_account":true},"enabled":true,"authentication_realm":{"name":"_service_account","type":"_service_account"},"lookup_realm":{"name":"_service_account","type":"_service_account"},"authentication_type":"token"}

• Do you have a separate monitoring cluster configured (using monitoring.ui.elasticsearch.hosts )?

Nope - we do not have a monitoring cluster.

[Yang_Wang]

@rsaeks The service token generated by elasticsearch-service-tokens CLI works only on the single node where the CLI runs. That is, the token will not work if it is used to talk to a different node of the same cluster. That is why you got the authentication error.

There are some gaps in our docs and upgrade guides where we are not clear about how the CLI works and should be used. I raised https://github.com/elastic/elasticsearch/issues/83491 to track it.

In summary, the CLI is more geared to be used as part of an orchestration flow. So what you roughly need is to generate the token on a single node, then copy the service_tokens file to all other nodes in your cluster. Alternatively, you can use the CreateServiceTokens API to generate a token that is backed by the security index and automatically usable on all nodes.

[rsaeks]

Thank you so much fore clarifying. When using the API to create the service tokens all worked as expected! I also put the feedback on the github issue. I hope this helps others out