Service Account Unable to Authenticate

Hello,

I have a healthy single node elasticsearch cluster but when I created a service token and plugged it into the kibana.yml file, I'm receiving an authentication error.

{
  "name" : "es01",
  "cluster_name" : "SANITIZED",
  "cluster_uuid" : "G2asaoshR8Sqg7LNN_B-ng",
  "version" : {
    "number" : "8.15.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1a77947f34deddb41af25e6f0ddb8e830159c179",
    "build_date" : "2024-08-05T10:05:34.233336849Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

The same is true with the curl command since I wanted to verify it wasnt my screwing up the config file.

{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [kibana]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [kibana]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

Did I miss something for the setup of the service account instead of using username/password as auth? I'm using the autogenerated/self-signed HTTPS certs if that matters.

SERVICE_TOKEN elastic/kibana/kibana = AAEAAWVsY....bDBHNkhHaFNyNjd0Z3ZjX1U5alN3
elasticsearch.hosts: ["https://X.X.3.15:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
elasticsearch.serviceAccountToken: "AAEAAWV...HNkhHaFNyNjd0Z3ZjX1U5alN3"

edit: verification of both elastic and kibana being on the same version-
"serviceVersion":{"source":"start","value":"8.15.0","commonName":"service_version"}

edit 2: after restarting elasticsearch, i found it was unable to start due to an issue reading the service tokens. i had to delete the token for it to start again.

	at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:72) ~[?:?]
	at org.elasticsearch.xpack.security.Security.createComponents(Security.java:1004) ~[?:?]
	at org.elasticsearch.xpack.security.Security.createComponents(Security.java:740) ~[?:?]
	... 18 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/service_tokens
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:379) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:431) ~[?:?]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
	at java.nio.file.Files.newInputStream(Files.java:159) ~[?:?]
	at java.nio.file.Files.newBufferedReader(Files.java:2902) ~[?:?]
	at java.nio.file.Files.readAllLines(Files.java:3397) ~[?:?]
	at org.elasticsearch.xpack.security.support.FileLineParser.parse(FileLineParser.java:21) ~[?:?]
	at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.parseFile(FileServiceAccountTokenStore.java:156) ~[?:?]
	at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:70) ~[?:?]
	at org.elasticsearch.xpack.security.Security.createComponents(Security.java:1004) ~[?:?]
	at org.elasticsearch.xpack.security.Security.createComponents(Security.java:740) ~[?:?]
	... 18 more
root@es01:~# ls /etc/elasticsearch/
certs  elasticsearch.keystore  elasticsearch-plugins.example.yml  elasticsearch.yml  jvm.options  jvm.options.d  log4j2.properties  role_mapping.yml  roles.yml  service_tokens  users  users_roles
root@es01:~# ls /etc/elasticsearch/ -la
total 72
drwxr-s---  4 root elasticsearch  4096 Aug 11 02:08 .
drwxr-xr-x 97 root root           4096 Aug 10 17:37 ..
drwxr-x---  2 root elasticsearch  4096 Aug  9 20:46 certs
-rw-rw----  1 root elasticsearch   536 Aug  9 20:46 elasticsearch.keystore
-rw-rw----  1 root elasticsearch  1042 Aug  5 10:08 elasticsearch-plugins.example.yml
-rw-rw----  1 root elasticsearch  4057 Aug 11 01:43 elasticsearch.yml
-rw-rw----  1 root elasticsearch  2781 Aug  5 10:08 jvm.options
drwxr-s---  2 root elasticsearch  4096 Aug  5 10:13 jvm.options.d
-rw-rw----  1 root elasticsearch 17969 Aug  5 10:08 log4j2.properties
-rw-rw----  1 root elasticsearch   473 Aug  5 10:08 role_mapping.yml
-rw-rw----  1 root elasticsearch   197 Aug  5 10:08 roles.yml
-rw-------  1 root elasticsearch   134 Aug 11 02:08 service_tokens
-rw-rw----  1 root elasticsearch   318 Aug 11 02:07 users
-rw-rw----  1 root elasticsearch    79 Aug 11 02:07 users_roles
root@es01:~# cat /etc/elasticsearch/service_tokens 
elastic/kibana/kibana:{PBKDF2_STRETCH}10000$/hHYco...ZgMBAPb6Gzij/F+9U

Bump for activity. I still haven't been able to resolve this yet for some reason.

Did you check the permissions on this file?

I don't believe I created that file but I'll double check later tonight and report back. Thanks for the tip!

Hey, I deleted the 1 token I had created previously, verified the service_tokens file didnt exist, then created another one. The file is equiv to 600; screenshot for reference.

Changing the perms to 660 allows me to authenticate with the curl command but for some reason has an issue when implemented in kibana.

Can you show the exact command you used to create the service token.... In txt please...

What do the Kibana and Elasticsearch logs logs show? When starting Kibana

Apologies, I thought I had included that in my first post to be complete with info.

Here's what worked for me yesterday.

curl -H "Authorization: Bearer AAEAAWVs...a2liYW5hL2tpYmFuYV90b2tlbjo3UlRGSVBieVNKaUFsQzdCODhpWjhn" https://localhost:9200/_cat/health?v -k

I meant the command you used to create the token.

Also you need to look in the logs...

Oh, sorry I misunderstood you and gave you the curl command in its stead.

Here's what I used to create the token that had given permission errors both in Kibana and curl:
./elasticsearch-service-tokens create elastic/kibana kibana

While it still has issues within Kibana, renaming the token to kibana_token allowed me to auth via curl.

./elasticsearch-service-tokens create elastic/kibana kibana_token was the command to create that semi-working token.

EDIT: Disregard below... I had 1 character off in the serviceAccountToken line. Kibana works with it now and I was ultimately a user error. I'm not sure why the token didnt work when named kibana vs kibana_token but that did solve my problem.

Thank you for your assistance!

Here's the kibana log showing the auth error.

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-08-20T19:41:47.631+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception\n\tRoot causes:\n\t\tsecurity_exception: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":35678,"uptime":25.808453737},"trace":{"id":"616785b927c0064a686fd7ffebf53c5d"},"transaction":{"id":"253fd5933d6ff30d"}}`
1 Like

When you have eliminated everything else... typo ... Ugh...
Not the first and Certainly not the last.