Elasticsearch 7.8.0 user authentication with X-Pack security fails

Elastic Stack Elasticsearch
1

Hello,

I'm experiencing user authentication issues for custom user, roles with X-Pack security.
I'm using elasticsearch REST API for the user, roles, etc. setup (Not using Kibana).
Responses with errors when checking cluster health and indices with REST API:
...security_exception, "reason":"action [cluster:monitor/health] is unauthorized for user [estester]" ... "status":403
...security_exception, "reason":"action [indices:monitor/settings/get] is unauthorized for user [estester]" ... "status":403

Question: where and how these "action [cluster:monitor/health]" and "action [indices:monitor/settings/get]" need to be setup or configured ?

The details are below.

I installed elasticsearch 7.8.0, basic license (elasticsearch-7.8.0-linux-86_64.tar.gz) on RHEL 5.11
Initial configuration: 2 node cluster for poc; one is master, one is data; xpack security disabled in es .yml file.
This initial es cluster setup worked well. With No security setup I was able to monitor cluster (health, etc.) and create/read/write/delete indices with curl REST API.

My ES security configuration with X-Pack and RBAC user setup
I ran command on master node
bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""
and file elastic-certificates.p12 has been generated in .../config folder
I copied it to .../config folder of the data node

I enabled security in es .yml on the master and data nodes
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Generated passwords for the built-in users (elastic, etc.) by running the following command on master node
bin/elasticsearch-setup-passwords auto
Restarted nodes.

I was able to use REST API to monitor the cluster health, etc. successfully as "elastic" superuser, like
curl -u elastic:el_pwd -XGET "host:port/_cluster/health"

Created new new custom role "tester_role" with certain privileges
curl -u elastic:el_pwd -XPOST "host:port/_security/role/tester_role" -H 'Content-Type: application/json' -d'
{"cluste{"cluster": ["monitor","manage_index_templates"],
"indices"indices": [{"names": ["*"], "priviliges": ["all"]}]}'
Created new custom user "estester" (using native realm) and than assigned the role "tester_role" to it
curl -u elastic:el_pwd -XPOST "host:port/_security/role_mapping/mapping1" -H 'Content-Type: application/json' -d'
{
"roles": ["tester_role"],
"enabled": true,
"rules": {"field": { "username": "estester"}}
}'

Checked new user, role and privileges assigned to role, everything were as expected.
Issue 1 when checking cluster health:
curl -u estester:estester_pwd -XGET "host:port/_cluster/health?pretty"
response: { error... root_cause.. security_exception, "reason":"action [cluster:monitor/health] is unauthorized for user [estester]" ... "status":403

Issue 2 when checking indices:
curl -u estester:estester_pwd -XGET "host:port/_cat/indices?h=index"
response: { error... root_cause.. security_exception, "reason":"action [indices:monitor/settings/get] is unauthorized for user [estester]" ... "status":403

I looked at the REST API security docs, did some reserach and not found any specifics/bugs/issues in regards to my issue.
Any ideas, hints, prompts what might be done to address the issue will be appreciated.

Thanks in advance

2

I addressed the issues. Whatever steps I proceeded were correct.
The problem was because I initially created the user with particular role name (not having it yet at that moment). Later on I created the role with different name (not equal to which I used during user creation) and assigned it to the user and that is why I got "not authenticated user". When I updated the user with the correct role's name - the problem disappeared and I was getting the expected responses for requests authentications, cluster and indices monitoring. The cluster is functional now with the X-Pack security enabled, RBAC user authentication, TLS for in-bound communication between cluster's nodes.

3

I'll need to setup SSL for out-bound communication between the cluster and java app client too.

4

I added SSL to config file elasticsearch.yml on two nodes as follows:

#security
xpack.security.enabled: true
#TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
#SSL
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

Restarted nodes.
Ran curl
curl -u estester:estester_pwd -XGET "https://host:port/_cat/indices?h=index"
and got response:
error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Elasticsearch log:
...javax.net.ssl.SSLHandshakeException: ClientRequestedProtocol TLSv1 not enabled or not supported

Does anyone know the curl format to test TLS/SSL secured cluster ?

5

Hello @dzyubanv,

javax.net.ssl.SSLHandshakeException: ClientRequestedProtocol TLSv1 not enabled or not supported

TLSv1 is obsolete. out of the box Elasticsearch does not support it https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#transport-tls-ssl-settings . You probably need to upgrade curl (or openssl / nss) .

Alternatively, but discouraged, you could set: xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"].

6

Thanks for the quick response Albert.
(a) I added the line 'xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]' to the config file on nodes and restarted them.
Ran curl -u estester:estester_pwd -XGET "http s ://host:port/_cat/indices?h=index" and got the same "javax.net.ssl.SSLHandshakeException: ClientRequestedProtocol TLSv1 not enabled or not supported" in the log.

(b) Also, I tried to test authentication as follows
curl --cert cert.crt --key cert.key --cacert cert.crt "http s ://host:port/_xpack/security/_authenticate?pretty"
and got on the following response on the console
unable to use client certificate (no key found or wrong pass phrase)

I'm trying to test somehow TLS/SSL of my secure cluster and make sure it is working as it should BEFORE I'll write and test simple java app client. I tested TLS part with curl successfully and now experiencing the issue with SSL testing. Also not sure why you said "TLSv1 is obsolete" when it is listed in docs for v.7.9 ? I'm using es v.7.8.0
Any advise how to workarount this issue ? thanks in advance

7

Ooops, sorry, it should be http not transport.

xpack.security.http.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

Also not sure why you said "TLSv1 is obsolete" when it is listed in docs for v.7.9 ? I'm using es v.7.8.0

TLS 1.0 is 21 years old RFC 2246 - The TLS Protocol Version 1.0 . It doesn't support the latest ciphers.

8

Replaced ".transport." with ".http." in es yml and restarted nodes
Ran curl -u estester:estester_pwd -XGET "https://host:port/_cat/indices?h=index"
Es log:
... io.netty..handler.codec..DecoderException: javax.net.ssl.SSLException: Received fatal alert: uncnown.ca
Response on console:
SSL certificate problem, verify that the CA cert is ok. Details:
error : SSL routines: SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). The default bundle is named curl-ca-bundle.crt; you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.

9

I also ran
curl --cacert certs/elastic-certificates.p12 -u estester:estester_pwd -XGET "http s ://host:port/_cat/indices?h=index"
Response on the console:
curl: (60) error setting certificate verify location:
CAfile: certs/elastic-certificates.p12
CApath: none
curl performs SSL certificate verification ... (the same as before)

I'm not sure what this CAfile/CApath means here (?)

10 
       --cacert <CA certificate>
              (TLS)  Tells  curl  to  use  the specified certificate file to verify the peer. The file may contain multiple CA certificates. The certificate(s) must be in PEM format.

The curl option requires the cert in a PEM format but you're passing the cert in a PKCS12 format.

You can convert it with something similar to:

openssl pkcs12 -in certs/elastic-certificates.p12 -out certs/elastic-certificates.crt -nokeys

But be advised that, from my reading of the file names, you're passing in client certificates as certificate authorities.

Since the conversation strayed away from the original topic, maybe it's worth to open another thread, or better still find something that applies to your present situation, such as Mutual tls/ssl on elasticsearch - #17 by SukeshGupta .

11

I appreciate your patience and desire to help Albert and will follow up with the link, but let me clarify something.

I tried to use your command for file format conversion but it prompted me for pwd, I clicked enter (dont know the pwd) and empty .crt file has been genarated with zero size.
Generated ca.crt, ca.key, instance.crt, instance.key with
bin/elasticsearch-certutil cert ca --pem --keep-ca-key --out certificates/certs.zip
unzipped and can see 4 files ca.crt, ca.key, instance.crt, instance.key
I didn't use the --in option. Is it ok ?

Q1: I configured 2 node es cluster with the TLS security section in es yml file using certificates in PKCS#12 format, native realm by default; created custom role, assigned privileges to it; created custom user/pwd, assigned the role to the user; was able to successfully test the cluster with the curl command (as above).
After then I added the SSL security section to es yml file (as above) using using the same certificates in PKCS#12 format.
When it came for testing the TLS/SSL secured cluster (both TLS and SSL security sections are presented in es yml file) with the curl it became obvious that the certificates must be generated and appear in es config yml file in PEM format rather then in PKCS#12 and for further java client testing and creating java keystore having the certificates in PEM format is mandatory.
Am I correct or missing something ?

Thanks in advance

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.