Service Tokens Replication Across Nodes and Kibana Integration

Elastic Search
1

Hello Elastic Community,

I am exploring the use of service tokens for connecting Kibana to a 3-node Elasticsearch cluster with TLS/SSL enabled. I have observed the following behavior and would like some clarification and guidance:

Cluster setup:

  • Elasticsearch 8.6.2, 3 nodes: elasticsearch, elasticsearch-01, elasticsearch-02

  • TLS enabled (xpack.security.transport.ssl.enabled: true)

  • Kibana 8.6.2 connecting via HTTPS

Steps I tried with service tokens:

  1. Created a service token on one node:

bin/elasticsearch-service-tokens create elastic/kibana kibana-server2

Output:

SERVICE_TOKEN elastic/kibana/kibana-server2 = <token_value>

  1. Checked the service token on all nodes using:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt
-u elastic:
-X GET "``https://elasticsearch:9200/_security/service/elastic/kibana/credential?pretty``"

Result:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt -u elastic:test   https://elasticsearch:9200/_security/service/elastic/kibana/credential?pretty
{
  "service_account" : "elastic/kibana",
  "count" : 2,
  "tokens" : { },
  "nodes_credentials" : {
    "_nodes" : {
      "total" : 3,
      "successful" : 3,
      "failed" : 0
    },
    "file_tokens" : {
      "kibana-token" : {
        "nodes" : [
          "elasticsearch"
        ]
      },
      "kibana-server2" : {
        "nodes" : [
          "elasticsearch-02"
        ]
      }
    }
  }
}
  1. Attempting to use the token on other nodes fails:
curl -k -H "Authorization: Bearer <token_value>" \
  "https://elasticsearch-01:9200/_security/_authenticate?pretty"

Result:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "failed to authenticate service account [elastic/kibana] with token name [kibana-server2]",
        "header" : {
          "WWW-Authenticate" : [
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "Bearer realm=\"security\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "failed to authenticate service account [elastic/kibana] with token name [kibana-server2]",
    "header" : {
      "WWW-Authenticate" : [
        "Basic realm=\"security\" charset=\"UTF-8\"",
        "Bearer realm=\"security\"",
        "ApiKey"
      ]
    }
  },
  "status" : 401
}

Cluster status:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt -u elastic:password \
>   https://elasticsearch:9200/_cluster/health?pretty
{
  "cluster_name" : "prod-es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt -u elastic:pass \
>   https://elasticsearch:9200/_cat/nodes?v
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.20.0.3           19          98   0    2.04    2.39     2.49 cdfhilmrstw -      elasticsearch
172.20.0.2           50          98   0    2.04    2.39     2.49 cdfhilmrstw -      elasticsearch-01
172.20.0.4           57          98   0    2.04    2.39     2.49 cdfhilmrstw *      elasticsearch-02

Observations:
• The service token only works on the node where it was created.
• There is no file in the Elasticsearch data/ directories that can be copied to other nodes.
• This makes it impossible to have Kibana connect to multiple nodes via a single service token if the cluster has multiple nodes.

Questions:

  1. Is there any way to replicate service tokens across all nodes in an Elasticsearch cluster?
  2. If service tokens are intentionally node-local, what is the recommended approach to allow Kibana to connect securely to the cluster without using a username/password?
  3. Are there any best practices for setting up Kibana with service tokens or API keys in multi-node clusters with TLS enabled?

Any guidance or examples would be greatly appreciated!

2

Resolved
Need to use for fix

1 Like
3

Hi @Somalic Welcome to the community.

Thanks for posting a well formed topic and then providing an answer.

But apologies I looked at the docs and I did not see the obvious difference/ fix could you provide a little more clarification on the fix?

Others may fine it useful.

Thanks