Service Tokens Replication Across Nodes and Kibana Integration

Elastic Search
1

Hello Elastic Community,

I am exploring the use of service tokens for connecting Kibana to a 3-node Elasticsearch cluster with TLS/SSL enabled. I have observed the following behavior and would like some clarification and guidance:

Cluster setup:

  • Elasticsearch 8.6.2, 3 nodes: elasticsearch, elasticsearch-01, elasticsearch-02

  • TLS enabled (xpack.security.transport.ssl.enabled: true)

  • Kibana 8.6.2 connecting via HTTPS

Steps I tried with service tokens:

  1. Created a service token on one node:

bin/elasticsearch-service-tokens create elastic/kibana kibana-server2

Output:

SERVICE_TOKEN elastic/kibana/kibana-server2 = <token_value>

  1. Checked the service token on all nodes using:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt
-u elastic:
-X GET "``https://elasticsearch:9200/_security/service/elastic/kibana/credential?pretty``"

Result:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt -u elastic:test   https://elasticsearch:9200/_security/service/elastic/kibana/credential?pretty
{
  "service_account" : "elastic/kibana",
  "count" : 2,
  "tokens" : { },
  "nodes_credentials" : {
    "_nodes" : {
      "total" : 3,
      "successful" : 3,
      "failed" : 0
    },
    "file_tokens" : {
      "kibana-token" : {
        "nodes" : [
          "elasticsearch"
        ]
      },
      "kibana-server2" : {
        "nodes" : [
          "elasticsearch-02"
        ]
      }
    }
  }
}
  1. Attempting to use the token on other nodes fails:
curl -k -H "Authorization: Bearer <token_value>" \
  "https://elasticsearch-01:9200/_security/_authenticate?pretty"

Result:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "failed to authenticate service account [elastic/kibana] with token name [kibana-server2]",
        "header" : {
          "WWW-Authenticate" : [
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "Bearer realm=\"security\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "failed to authenticate service account [elastic/kibana] with token name [kibana-server2]",
    "header" : {
      "WWW-Authenticate" : [
        "Basic realm=\"security\" charset=\"UTF-8\"",
        "Bearer realm=\"security\"",
        "ApiKey"
      ]
    }
  },
  "status" : 401
}

Cluster status:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt -u elastic:password \
>   https://elasticsearch:9200/_cluster/health?pretty
{
  "cluster_name" : "prod-es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt -u elastic:pass \
>   https://elasticsearch:9200/_cat/nodes?v
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.20.0.3           19          98   0    2.04    2.39     2.49 cdfhilmrstw -      elasticsearch
172.20.0.2           50          98   0    2.04    2.39     2.49 cdfhilmrstw -      elasticsearch-01
172.20.0.4           57          98   0    2.04    2.39     2.49 cdfhilmrstw *      elasticsearch-02

Observations:
• The service token only works on the node where it was created.
• There is no file in the Elasticsearch data/ directories that can be copied to other nodes.
• This makes it impossible to have Kibana connect to multiple nodes via a single service token if the cluster has multiple nodes.

Questions:

  1. Is there any way to replicate service tokens across all nodes in an Elasticsearch cluster?
  2. If service tokens are intentionally node-local, what is the recommended approach to allow Kibana to connect securely to the cluster without using a username/password?
  3. Are there any best practices for setting up Kibana with service tokens or API keys in multi-node clusters with TLS enabled?

Any guidance or examples would be greatly appreciated!

2

Resolved
Need to use for fix

1 Like
3

Hi @Somalic Welcome to the community.

Thanks for posting a well formed topic and then providing an answer.

But apologies I looked at the docs and I did not see the obvious difference/ fix could you provide a little more clarification on the fix?

Others may fine it useful.

Thanks

4

Sorry this is a very late reply - I don't always stay on top of posts in this forum.

The key information is this part of the docs:

The recommended way to manage service tokens is via the Create service account tokens API. File based tokens are intended for use with orchestrators such as Elastic Cloud Enterprise and Elastic Cloud on Kubernetes

Elasticsearch does not provide any mechanism to replicate the file based service tokens between nodes.

  • If you are using an orchestrator (such as ECE or ECK) then it will manage the file for you and apply it to all nodes.
  • If you are not using an orchestrator then you should use the Rest API to manage service tokens instead of the command line tool.
  • If you really want to use file based service tokens without using one of our official orchestration platforms, then you need to manage the orchestration yourself (e.g. you might bake the file into a docker image or an AMI so that it is automatically provisioned on new nodes).
1 Like