# Service token causes index\_not\_found\_exception

**URL:** <https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900>\
**Category:** Kibana\
**Tags:** elastic-stack-security\
**Created:** [October 5, 2022, 4:33pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900 "2022-10-05T16:33:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [October 5, 2022, 4:33pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/1 "2022-10-05T16:33:52Z")

</div>

I want to learn to use service tokens. I followed the instructions here:

> **[elasticsearch-service-tokens | Elasticsearch Guide \[8.4\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/service-tokens-command.html)**

Specifically, I ran this command from bash CLI of the server `139.177.199.119`

```auto
bin/elasticsearch-service-tokens create elastic/kibana my-token

```

Which gave this result:

```auto
SERVICE_TOKEN elastic/kibana/my-token = AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEtTm1IY1c4UTRxMDNUM2YzVUNOcFE

```

I then ran this command:

```auto
curl -X GET -k -H "Authorization: Bearer AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOnEtTm1IY1c4UTRxMDNUM2YzVUNOcFE" "https://139.177.199.119:9200/_security/_authenticate"

```

Which gave this result:

```auto
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.security]","resource.type":"index_expression","resource.id":".security","index_uuid":"_na_","index":".security"}],"type":"security_exception","reason":"error attempting to authenticate request","caused_by":{"type":"index_not_found_exception","reason":"no such index [.security]","resource.type":"index_expression","resource.id":".security","index_uuid":"_na_","index":".security"},"header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

```

What did I do wrong?

---

<div class="post-metadata">

**Author:** ![Yang\_Wang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yang_wang/32/48857_2.png) [@Yang\_Wang](https://discuss.elastic.co/u/Yang_Wang)\
**Post date:** [October 6, 2022, 2:30am UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/2 "2022-10-06T02:30:44Z")

</div>

Service tokens created using the CLI is per node, i.e. you can authenticate with it only to the node where you generated it. If you use it to talk to a different node, it won't work and you can observed the error if security index is also not created yet.

Please either make sure you talk to the same node that generated the token or copy the `service_tokens` file to all nodes.

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [October 6, 2022, 12:51pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/3 "2022-10-06T12:51:42Z")

</div>

Thank you Yang

I only one have server `139.177.199.119` which I issued all my CLI commands from. Does that meet your definition of the same node?

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [October 6, 2022, 1:26pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/4 "2022-10-06T13:26:34Z")

</div>

Also now the command `systemctl restart elasticsearch.service` fails to start up the elastic service. My `/var/log/elasticsearch/my-application.log` shows this:

```auto
[2022-10-06T13:23:16,729][ERROR][o.e.b.Elasticsearch] [localhost] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: security initialization failed
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:562) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$15(Node.java:696) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:236) ~[elasticsearch-8.4.3.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:710) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.3.jar:?]
Caused by: java.lang.IllegalStateException: Failed to load service_tokens file [/etc/elasticsearch/service_tokens]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:72) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:742) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:550) ~[?:?]
        ... 17 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/service_tokens
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:380) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:432) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at java.nio.file.Files.newBufferedReader(Files.java:2923) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3413) ~[?:?]
        at org.elasticsearch.xpack.security.support.FileLineParser.parse(FileLineParser.java:21) ~[?:?]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.parseFile(FileServiceAccountTokenStore.java:156) ~[?:?]
        at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.<init>(FileServiceAccountTokenStore.java:70) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:742) ~[?:?]
        at org.elasticsearch.xpack.security.Security.createComponents(Security.java:550) ~[?:?]
        ... 17 more

```

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [October 6, 2022, 2:04pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/5 "2022-10-06T14:04:33Z")

</div>

I figured it out!

I needed to run the following commands:

```auto
chmod 0644 /etc/elasticsearch/service_tokens

ls -alh /etc/elasticsearch/service_tokens
>>> -rw-r--r-- 1 root elasticsearch 136 Oct 6 13:58 service_tokens

systemctl restart elasticsearch.service

```

Now my curl statement works and i get this response:

```auto
"https://139.177.199.119:9200/_cluster/health?wait_for_status=yellow&timeout=50s"
{"cluster_name":"my-application","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

```

---

<div class="post-metadata">

**Author:** ![Yang\_Wang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yang_wang/32/48857_2.png) [@Yang\_Wang](https://discuss.elastic.co/u/Yang_Wang)\
**Post date:** [October 6, 2022, 11:42pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/6 "2022-10-06T23:42:44Z")

</div>

Thanks for providing your solution. In hindsight, you were hitting this issue [Prevent users from running elasticsearch-service-tokens as root · Issue #81378 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/81378)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 3, 2022, 11:43pm UTC](https://discuss.elastic.co/t/service-token-causes-index-not-found-exception/315900/7 "2022-11-03T23:43:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.