Kibana 8.13.4 Fails to Authenticate to Elasticsearch 8.13.4 with Service Account Token (security_exception)

Elastic Stack Kibana
1

Description:
I am running Elasticsearch and Kibana, both version 8.13.4, on Kubernetes. Kibana is configured to use a service account token (generated for elastic/kibana) mounted as a file and referenced in kibana.yml. Despite following all best practices, Kibana fails to authenticate with the following error:

[elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
Root causes:
    security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request

What I’ve validated:

  • The service account token is generated with:

    bin/elasticsearch-service-tokens create elastic/kibana kibana-service-account

  • The token is stored in a Kubernetes secret and mounted as a file in the Kibana pod.

  • The token file has no trailing newline (checked with od -c).

  • The token is valid:
    Running curl -k -H "Authorization: Bearer $(cat /usr/share/kibana/config/serviceAccountToken)" https://elasticsearch:9200 from inside the Kibana pod returns cluster info.

  • kibana.yml contains only:

    elasticsearch.hosts: ["https://elasticsearch:9200"]
elasticsearch.ssl.verificationMode: none
elasticsearch.serviceAccountToken: /usr/share/kibana/config/serviceAccountToken

  • No extra authentication environment variables are set in the pod.

  • Both Elasticsearch and Kibana are version 8.13.4.

  • I have tried regenerating the token, using a different token name, and restarting the pod.

  • There are no custom security plugins or network policies interfering.

Kibana Deployment YAML (excerpt):

        volumeMounts:
        - name: kibana-service-token
          mountPath: /usr/share/kibana/config/serviceAccountToken
          subPath: serviceAccountToken
        - name: kibana-config
          mountPath: /usr/share/kibana/config/kibana.yml
          subPath: kibana.yml
      volumes:
      - name: kibana-service-token
        secret:
          secretName: kibana-service-token
      - name: kibana-config
        configMap:
          name: kibana-config

What else can I try, or is this a known issue?
Any help or insight would be greatly appreciated!

Let me know if you want anything further from the setup.

2

Hi @Himanshu_Sharma1

Welcome to the community.... sorry for the trouble.

Can you share where you saw this?

I don't think you can set the service token to a path to a file for Kibana. You can do things like that in Elasticsearch, but I do not think you can in Kibana, so try setting it directly in the kibana.yml, btw, this token is a low-risk token as it only allows Kibana to connect to Elasticsearch is does not allow any user actions in Kibana or Elastic

elasticsearch.serviceAccountToken: /usr/share/kibana/config/serviceAccountToken

I think you need to set it in the kibana.yml or set up the whole kibana spec in your deployment manifest and substitute in the secret using k8s secrets

All that said, curious if you are running Elasticsearch and Kibana on K8s, why you are not using ECK, where all this is done for you... That would be the recommended approach.