Environment:
- CentOS 7
- Active Directory
- Elasticsearch & Kibana 7.9.3
- Realms configured in ES: Kerberos and AD
- Kerberos authorization_realm: AD
ES cluster set up with keytabs on master and data nodes. I can run curl --negotiate -u : https://master-fqdn:9200/_security/_authenticate?pretty aginst all nodes and get a nice JSON describing the user and groups from AD. I can do this from any Windows or Linux host where I have a Kerberos session.
Kibana is setup with only one provider:
xpack.security.authc.providers:
kerberos.kerberos1:
order: 0
description: "Log in with kerberos"
Kibana server is enrolled in kerberos domain (all machines are). Time sync and reverse lookup is working.
When using Firefox to access Kibana (after setting network.negotiate-auth.trusted-uris in about:config) on both Linux and Windows it sends the Authorization: Negotiate header with the SPNEGO data, just like curl does. But here I get GSSException: Defective token detected. GSSHeader did not find the right tag in the ES log and a 401 Unauthorized from Kibana. In the Kibana log I get
SPNEGO is supported by the backend
Re-initiating SPNEGO handshake
Authentication attempt failed: Unauthorized
I have activated krb5 and spnego debug with JVM command line options for ES. When I run curl --negotiate I see a lot of Kerberos output in the log, but it's silent when the Kibana reuqest fails. I only see the GSSException stacktrace in the ES log. It seems to me it fails early when trying to extract the SPNEGO token?
It's not NTLM (common error when you see "Defective token") since I get same error on Firefox@Linux and the token looks similar to the one I get with curl.
Is it possible to log the raw http requests coming into ES? It's TLS so not easy to tcpdump the traffic from Kibana to ES. I guess what should happen is that Kibana forwards the Authorization header to ES.
Any other debugging steps to take?