Service token causes index_not_found_exception

learningelastic · October 5, 2022, 4:33pm

I want to learn to use service tokens. I followed the instructions here:

Specifically, I ran this command from bash CLI of the server 139.177.199.119

bin/elasticsearch-service-tokens create elastic/kibana my-token

Which gave this result:

SERVICE_TOKEN elastic/kibana/my-token = AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEtTm1IY1c4UTRxMDNUM2YzVUNOcFE

I then ran this command:

curl -X GET -k -H "Authorization: Bearer AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEtTm1IY1c4UTRxMDNUM2YzVUNOcFE" "https://139.177.199.119:9200/_security/_authenticate"

Which gave this result:

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.security]","resource.type":"index_expression","resource.id":".security","index_uuid":"_na_","index":".security"}],"type":"security_exception","reason":"error attempting to authenticate request","caused_by":{"type":"index_not_found_exception","reason":"no such index [.security]","resource.type":"index_expression","resource.id":".security","index_uuid":"_na_","index":".security"},"header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

What did I do wrong?

Yang_Wang · October 6, 2022, 2:30am

Service tokens created using the CLI is per node, i.e. you can authenticate with it only to the node where you generated it. If you use it to talk to a different node, it won't work and you can observed the error if security index is also not created yet.

Please either make sure you talk to the same node that generated the token or copy the service_tokens file to all nodes.

learningelastic · October 6, 2022, 12:51pm

Thank you Yang

I only one have server 139.177.199.119 which I issued all my CLI commands from. Does that meet your definition of the same node?

learningelastic · October 6, 2022, 1:26pm

Also now the command systemctl restart elasticsearch.service fails to start up the elastic service. My /var/log/elasticsearch/my-application.log shows this:

[2022-10-06T13:23:16,729][ERROR][o.e.b.Elasticsearch      ] [localhost] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: security initialization failed
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:562) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$15(Node.java:696) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:236) ~[elasticsearch-8.4.3.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:710) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.3.jar:?]
Caused by: java.lang.IllegalStateException: Failed to load service_tokens file [/etc/elasticsearch/service_tokens]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:72) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:742) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:550) ~[?:?]
        ... 17 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/service_tokens
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:380) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:432) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at java.nio.file.Files.newBufferedReader(Files.java:2923) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3413) ~[?:?]
        at org.elasticsearch.xpack.security.support.FileLineParser.parse(FileLineParser.java:21) ~[?:?]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.parseFile(FileServiceAccountTokenStore.java:156) ~[?:?]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:70) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:742) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:550) ~[?:?]
        ... 17 more

learningelastic · October 6, 2022, 2:04pm

I figured it out!

I needed to run the following commands:

chmod 0644 /etc/elasticsearch/service_tokens

ls -alh /etc/elasticsearch/service_tokens
>>> -rw-r--r--   1 root elasticsearch  136 Oct  6 13:58 service_tokens

systemctl restart elasticsearch.service

Now my curl statement works and i get this response:

"https://139.177.199.119:9200/_cluster/health?wait_for_status=yellow&timeout=50s"
{"cluster_name":"my-application","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

Yang_Wang · October 6, 2022, 11:42pm

Thanks for providing your solution. In hindsight, you were hitting this issue Prevent users from running elasticsearch-service-tokens as root · Issue #81378 · elastic/elasticsearch · GitHub

system · November 3, 2022, 11:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.