Service token causes index_not_found_exception

I want to learn to use service tokens. I followed the instructions here:

Specifically, I ran this command from bash CLI of the server 139.177.199.119

bin/elasticsearch-service-tokens create elastic/kibana my-token

Which gave this result:

SERVICE_TOKEN elastic/kibana/my-token = AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEtTm1IY1c4UTRxMDNUM2YzVUNOcFE

I then ran this command:

curl -X GET -k -H "Authorization: Bearer AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEtTm1IY1c4UTRxMDNUM2YzVUNOcFE" "https://139.177.199.119:9200/_security/_authenticate"

Which gave this result:

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.security]","resource.type":"index_expression","resource.id":".security","index_uuid":"_na_","index":".security"}],"type":"security_exception","reason":"error attempting to authenticate request","caused_by":{"type":"index_not_found_exception","reason":"no such index [.security]","resource.type":"index_expression","resource.id":".security","index_uuid":"_na_","index":".security"},"header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

What did I do wrong?

Service tokens created using the CLI is per node, i.e. you can authenticate with it only to the node where you generated it. If you use it to talk to a different node, it won't work and you can observed the error if security index is also not created yet.

Please either make sure you talk to the same node that generated the token or copy the service_tokens file to all nodes.

Thank you Yang

I only one have server 139.177.199.119 which I issued all my CLI commands from. Does that meet your definition of the same node?

Also now the command systemctl restart elasticsearch.service fails to start up the elastic service. My /var/log/elasticsearch/my-application.log shows this:

[2022-10-06T13:23:16,729][ERROR][o.e.b.Elasticsearch      ] [localhost] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: security initialization failed
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:562) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$15(Node.java:696) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:236) ~[elasticsearch-8.4.3.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:710) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.3.jar:?]
Caused by: java.lang.IllegalStateException: Failed to load service_tokens file [/etc/elasticsearch/service_tokens]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:72) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:742) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:550) ~[?:?]
        ... 17 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/service_tokens
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:380) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:432) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at java.nio.file.Files.newBufferedReader(Files.java:2923) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3413) ~[?:?]
        at org.elasticsearch.xpack.security.support.FileLineParser.parse(FileLineParser.java:21) ~[?:?]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.parseFile(FileServiceAccountTokenStore.java:156) ~[?:?]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:70) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:742) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:550) ~[?:?]
        ... 17 more

I figured it out!

I needed to run the following commands:

chmod 0644 /etc/elasticsearch/service_tokens

ls -alh /etc/elasticsearch/service_tokens
>>> -rw-r--r--   1 root elasticsearch  136 Oct  6 13:58 service_tokens

systemctl restart elasticsearch.service

Now my curl statement works and i get this response:

"https://139.177.199.119:9200/_cluster/health?wait_for_status=yellow&timeout=50s"
{"cluster_name":"my-application","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

Thanks for providing your solution. In hindsight, you were hitting this issue Prevent users from running elasticsearch-service-tokens as root · Issue #81378 · elastic/elasticsearch · GitHub

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.