Defective token when using SPNEGO in Kibana

Elastic Stack Kibana
1

Environment:

  • CentOS 7
  • Active Directory
  • Elasticsearch & Kibana 7.9.3
  • Realms configured in ES: Kerberos and AD
  • Kerberos authorization_realm: AD

ES cluster set up with keytabs on master and data nodes. I can run curl --negotiate -u : https://master-fqdn:9200/_security/_authenticate?pretty aginst all nodes and get a nice JSON describing the user and groups from AD. I can do this from any Windows or Linux host where I have a Kerberos session.

Kibana is setup with only one provider:

xpack.security.authc.providers:
  kerberos.kerberos1:
    order: 0
    description: "Log in with kerberos"

Kibana server is enrolled in kerberos domain (all machines are). Time sync and reverse lookup is working.

When using Firefox to access Kibana (after setting network.negotiate-auth.trusted-uris in about:config) on both Linux and Windows it sends the Authorization: Negotiate header with the SPNEGO data, just like curl does. But here I get GSSException: Defective token detected. GSSHeader did not find the right tag in the ES log and a 401 Unauthorized from Kibana. In the Kibana log I get

SPNEGO is supported by the backend
Re-initiating SPNEGO handshake
Authentication attempt failed: Unauthorized

I have activated krb5 and spnego debug with JVM command line options for ES. When I run curl --negotiate I see a lot of Kerberos output in the log, but it's silent when the Kibana reuqest fails. I only see the GSSException stacktrace in the ES log. It seems to me it fails early when trying to extract the SPNEGO token?

It's not NTLM (common error when you see "Defective token") since I get same error on Firefox@Linux and the token looks similar to the one I get with curl.

Is it possible to log the raw http requests coming into ES? It's TLS so not easy to tcpdump the traffic from Kibana to ES. I guess what should happen is that Kibana forwards the Authorization header to ES.

Any other debugging steps to take?

2

Do your keytabs contain a service principal for the kibana hostname ? (Assuming kibana is on a different host or using a different hostname )

3

Thank you! That was it.

Somewhere I read that it is recommended to have a coordinating ([data, master, ingest]=false). If I had that the issue wouldn't arise, since there would be a node with the same SPN as Kibana host. Is this the recommended way?

4

It's recommended to have a coordinating node but not necessarily on the same host as Kibana.

The main issue here is that our docs do not point it out clearly enough. We are tracking this work in this kibana issue

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.