Elasticsearch 6.4 Kerberos: Checksum failed


(Constantine Abayeu) #1

Hello, there.

We are doing PoC on 6.4 build in Kerberos Auth for elasticsearch.

Here is a problem I have ran into with trying to set it up (please help me understand what is happening here):

my Kerberos part of elasticsearch.yml file

xpack.security.authc.realms:
    realm0:
        type: kerberos
        order: 0
        keytab.path: /etc/elasticsearch/client/SUPER_SECRET_ID.ktf
        krb.debug: true
        enabled: true
        remove_realm_name: false

I can not share my krb5.conf file, that is also in my elasticsearch directory, but that krb5 works fine with other applications, so I do not think the issue is in it.

Essentially here is the problem (that is the error in http client):

{
	"error": {
		"root_cause": [{
			"type": "security_exception",
			"reason": "Failure unspecified at GSS-API level (Mechanism level: Checksum failed)",
			"header": {
				"WWW-Authenticate": ["Basic realm=\"security\" charset=\"UTF-8\"", "Negotiate", "Bearer realm=\"security\""]
			}
		}],
		"type": "security_exception",
		"reason": "Failure unspecified at GSS-API level (Mechanism level: Checksum failed)",
		"caused_by": {
			"type": "g_s_s_exception",
			"reason": "Failure unspecified at GSS-API level (Mechanism level: Checksum failed)",
			"caused_by": {
				"type": "krb_crypto_exception",
				"reason": "Checksum failed",
				"caused_by": {
					"type": "general_security_exception",
					"reason": "Checksum failed"
				}
			}
		},
		"header": {
			"WWW-Authenticate": ["Basic realm=\"security\" charset=\"UTF-8\"", "Negotiate", "Bearer realm=\"security\""]
		}
	},
	"status": 401
}

and this is my elasticsearch log entry when attempt at auth happens:

[2018-08-30T10:32:33,663][INFO ][o.e.x.s.a.AuthenticationService] [SERVER_NAME-client] Authentication of [<Kerberos Token>] was terminated by realm [realm0] - failed to authenticate user, gss context negotiation failure

I have created the keytab file using the same ID we use for active_directory auth with elasticsearch that works just fine still.

and I am using aes256-sha1 as encryption for the addentry in ktutil.
Password has been now quadruple checked by multiple members of my team...


(Constantine Abayeu) #2

I have now tried multiple variations of keytab file...
Have tried setting it up with HTTP/ID@DOMAIN.COM
and without it...

here is the command I use in ktutil: addent -password -p HTTP/ID@DOMAIN.COM -k 1 -e aes256-sha1


(Constantine Abayeu) #3

Below is bunch of stack trace that I got down to now.

Basically from what it looks like, elasticsearch is having trouble deciphering the Kerberos Ticket?

So the thing is, though, when I do a TCP dump on that node, and look it over, it does not have ANY packets directed at any of the domain controllers that are specified in my krb5,conf file.

Should also mention, that I have attempted it with multiple different user set ups.

  1. logon name is "HTTP/servername.domain.com" + SPN set to "HTTP/servername"
  2. logon name is "HTTP/servername.domain.com" + SPN set to "HTTP/servername.domain.com"
  3. logon name is "HTTP/servername.domain.com" + SPN set to "HTTP/servername, HTTP/servername.domain.com"
  4. logon name is "HTTP/servername.domain.com" + no SPN set
  5. logon name is "elastictest1" and all the above SPN configurations.....

Below is the said stack trace....

[2018-08-31T10:28:29,933][DEBUG][o.e.x.s.a.k.KerberosRealm] [SERVER-client] failed to authenticate user, gss context negotiation failure
org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
    at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856) ~[?:?]
    at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:?]
    at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:?]
    at sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:906) ~[?:?]
    at sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:556) ~[?:?]
    at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:?]
    at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:?]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidator.lambda$acceptSecContext$0(KerberosTicketValidator.java:142) ~[x-pack-security-6.4.0.jar:6.4.0]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_131]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidator.lambda$doAsWrapper$2(KerberosTicketValidator.java:170) ~[x-pack-security-6.4.0.jar:6.4.0]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidator.doAsWrapper(KerberosTicketValidator.java:170) ~[x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidator.acceptSecContext(KerberosTicketValidator.java:141) ~[x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidator.validateTicket(KerberosTicketValidator.java:90) ~[x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.authc.kerberos.KerberosRealm.authenticate(KerberosRealm.java:138) ~[x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:262) ~[x-pack-security-6.4.0.jar:6.4.0]
    <..... skipped a bunch because they are repeating......>
Caused by: java.security.GeneralSecurityException: Checksum failed
    at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:?]
    at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:?]
    at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:?]
    at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:?]
    at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:?]
    at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:?]
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:?]
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149) ~[?:?]
    at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108) ~[?:?]
    at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829) ~[?:?]
    ... 86 more
[2018-08-31T10:28:29,967][INFO ][o.e.x.s.a.AuthenticationService] [SERVER-client] Authentication of [<Kerberos Token>] was terminated by realm [realm0] - failed to authenticate user, gss context negotiation failure

please, help! T_T


(Brad Quarry) #4

Bump. Anyone have any thoughts here? I know a few folks experiencing this configuration problem.


#5

First of all, if you happen to use ktpass.exe on Windows - don't!. it is a buggy piece of garbage, likely not touched much since days of Windows NT 4.0 or so.
It has some arbitrary limits on characters in password and length - but it never reports any errors in such cases, simply generates useless keytabs. It also does not handle SPNs very well (or any slightly more complex keytabs) and sometimes even corrupts existing accounts during the keytab generation process...

What is more important, interactive password prompt when generating keytab in ktpass.exe DOES NOT WORK. If you are attempting to use it with a know password to some preexisting account this WILL fail and the resulting keytab will never work - it looks like the tool appends new line character to the password you type (or something like that). Solution: provide password in a non-interactive way (inline with the command), but this might still fail to produce functioning keytab thanks to the bugs mentioned above...

Luckly, there is a tool for handling all of that which actually works: msktutil (https://github.com/msktutil/msktutil).

Assuming you have got kerberos environment working on your linux box with a ticket (kinit) for the account which is permitted to make changes to AD, you can do the following:

1. Create Elastcisearch service account for Kerberos auth (+ generate keytab for it):

msktutil --create --use-service-account --keytab /tmp/elastic-service.keytab --account-name elastic-service \
--dont-expire-password -b "OU=Users,OU=CompanyA" --password "SuperSecurePassw0rdToServiceAccount" --description "ELK Service account"

2. Add SPN to the account:

msktutil update -s HTTP/elasticsearch.domain.local --use-service-account --user-creds-only --account-name elastic-service \
--keytab /tmp/elastic-service.keytab --dont-change-password -b "OU=Users,OU=CompanyA"

Additionally, assuming your Elastic API runs on default port 9200 (likely it does), you need SPN for that specific port too (or at least it was needed in my case):

msktutil update -s HTTP/elasticsearch.domain.local:9200 --use-service-account --user-creds-only --account-name elastic-service \
--keytab /tmp/elastic-service.keytab --dont-change-password -b "OU=Users,OU=CompanyA"

After running all these commands, the resulting keytab in /tmp/elastic-service.keytab should contain entries for service account + all required SPNs with the right crypto flavours, etc.

You can easily test if the keytab is in fact any good by using kinit:

kinit -kt /tmp/elastic-service.keytab elastic-service

and for SPN:

kinit -kt /tmp/elastic-service.keytab elastic-service -S HTTP/elasticsearch.domain.local
(try also for HTTP/elasticsearch.domain.local:9200)

After every command you should see that relevant ticket is present in keyring with klist

Hope this helps!


(Constantine Abayeu) #6

I was using ktutil on linux. all of our elastic things run on linux.

I bet it is a SPN thing, because I did not do it with the port on it. I will try it once more, when I get a test cluster running.

Thank you for your reply!


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.