After creating API key attempting to use it results in: missing authentication credentials for REST request

[scarby (Adam E Scarborough)]

Hope someone can help here

i'm attempting to create api keys for a user and utilize them as documented here:
https://www.elastic.co/guide/en/elasticsearch/reference/7.5/security-api-create-api-key.html

i have successfully created the api key however when i attempt to curl anything as documented in the elastic search docs i.e.:

curl -k https://<censored>:9200/_cluster/health -H 'Authorization: ApiKey <censored>='

i am greeted with the following response:

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cluster/health]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cluster/health]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

I'm not sure why it would complain here given that i have apparently valid api token.

After doing some more digging i've noticed that i'm getting an invalid credentials error logged from elastic-search - the credentials are in fact correct and were pulled from the response using the following:

echo $(cat response | jq -r .id):$(cat response | jq -r .api_key) | base64

Incidentally if i intentionally use the wrong key id (in this case by including quotes) the following is logged

Authentication using apikey failed - unable to find apikey with id "fufe928Bnl45U--l7GCn"

so seemingly elasticsearch is finding the key but failing to grant access for other reasons

[ikakavas (Ioannis Kakavas)]

Which version is this on @scarby , is this reproducible with other API keys too ?

[scarby (Adam E Scarborough)]

@ikakavas 7.5.0 - deployed using the kubernetes operator

and yes, i am not able to auth using any api key i have created. I have tried multiple - both with and without roledefinitions explicitly specified

[ikakavas (Ioannis Kakavas)]

@scarby your echo needs the -n flag, otherwise it adds a newline to the string to be base64 encoded and the produced API Key is wrong.

i.e. , assuming you have a response like

{"id":"uggYCnABMV990shiRGts","name":"myname","api_key":"g8x2HywBS2iS1jbmJv4pNw"}

Now,

echo $(cat response | jq -r .id):$(cat response | jq -r .api_key) | base64

gives you

dWdnWUNuQUJNVjk5MHNoaVJHdHM6Zzh4Mkh5d0JTMmlTMWpibUp2NHBOdwo=

while

echo -n $(cat response | jq -r .id):$(cat response | jq -r .api_key) | base64

gives you

dWdnWUNuQUJNVjk5MHNoaVJHdHM6Zzh4Mkh5d0JTMmlTMWpibUp2NHBOdw==

The latter is the ApiKey value you want and decodes to:

$ echo "dWdnWUNuQUJNVjk5MHNoaVJHdHM6Zzh4Mkh5d0JTMmlTMWpibUp2NHBOdw==" | base64 -d | cat -vE

uggYCnABMV990shiRGts:g8x2HywBS2iS1jbmJv4pNw

while the former decodes to

$ echo "dWdnWUNuQUJNVjk5MHNoaVJHdHM6Zzh4Mkh5d0JTMmlTMWpibUp2NHBOdwo=" | base64 -d | cat -vE

uggYCnABMV990shiRGts:g8x2HywBS2iS1jbmJv4pNw$

(mind the trailing $)

HTH