# Need help with X-pack TLS authentication **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [February 12, 2019, 10:40pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121 "2019-02-12T22:40:29Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jbourne](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jbourne](https://discuss.elastic.co/u/jbourne)\ **Post date:** [February 12, 2019, 10:40pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/1 "2019-02-12T22:40:29Z")
I'm trying to setup an elastic stack on kubernetes in AWS. I have: 1 - 3 master nodes 2 - 2 client nodes 3 - 1 data node In Kubernetes, I've created a Service that exposes ports 9200 and 9300. This service gets applied to an AWS ELB which creates listeners for those ports. The full DNS name for the ELB is ct-es.ct-es.svc.clsuter.local. The pods for their respective containers execute successfully but X-pack and PKI authentication is where I'm running into problems. When I try to execute a curl request shown below, I get a 401 error in the response: \_curl -X GET [https://ct-es.ct-es.svc.cluster.local:9200/](https://ct-es.ct-es.svc.cluster.local:9200/)_cluster/health?pretty --cacert /usr/share/elasticsearch/config/ca.crt_ ``` { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "missing authentication token for REST request [/_cluster/health?pretty]", "header" : { "WWW-Authenticate" : [ "Bearer realm=\"security\"", "Basic realm=\"security\" charset=\"UTF-8\"" ] } } ], "type" : "security_exception", "reason" : "missing authentication token for REST request [/_cluster/health?pretty]", "header" : { "WWW-Authenticate" : [ "Bearer realm=\"security\"", "Basic realm=\"security\" charset=\"UTF-8\"" ] } }, "status" : 401 } } ``` This is what my elasticsearch.yml file looks like on the client node: ``` cluster: name: logs xpack.license.self_generated.type: trial xpack.monitoring.enabled: true xpack.security.enabled: true xpack.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key xpack.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt xpack.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/ca.crt"] xpack.ssl.client_authentication: required xpack.ssl.verification_mode: certificate xpack.security.http.ssl.enabled: true xpack.security.http.ssl.verification_mode: certificate xpack.security.http.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt xpack.security.http.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/ca.crt"] xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt xpack.security.transport.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/ca.crt"] xpack.security.audit.enabled: true xpack.security.audit.outputs: [index, logfile] xpack.security.audit.index.settings: index: number_of_shards: 1 number_of_replicas: 1 xpack.security.authc.realms: realm1: type: native order: 0 realm2: type: pki order: 1 network.host: _eth0_ thread_pool.bulk.queue_size: 800 path: data: /usr/share/elasticsearch/data logs: /usr/share/elasticsearch/logs http: enabled: true compression: true ``` ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [February 13, 2019, 6:17am UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/2 "2019-02-13T06:17:03Z")
> [@jbourne](#): > > When I try to execute a curl request shown below, I get a 401 error in the response: > > \_curl -X GET [https://ct-es.ct-es.svc.cluster.local:9200/](https://ct-es.ct-es.svc.cluster.local:9200/) _cluster/health?pretty --cacert /usr/share/elasticsearch/config/ca.crt_ You have security enabled in Elasticsearch with 2 security realms (native and PKI) so you need to send some client credentials in your request. You need to send either - a username and password combination for a user that exists in the native realm , with `-uusername:password` - a client certificate by specifying `--cert` and `--key` , see the curl man page at [curl - How To Use](https://curl.haxx.se/docs/manpage.html) Please also read through our [PKI realm documentation](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/configuring-pki-realm.html), it will be really helpful. ---
**Author:** ![jbourne](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jbourne](https://discuss.elastic.co/u/jbourne)\ **Post date:** [February 13, 2019, 9:19pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/3 "2019-02-13T21:19:55Z")
I've looked at PKI realm documentation and it hasn't helped. I'd like authentication to happen using PKI because that removes the hassle of maintaining passwords. I tried providing a client certificate and the key but still keep getting the same result. My questions are: 1. What's this in the HTTP header request- > ``` > "header" : { > "WWW-Authenticate" : [ > "Bearer realm=\"security\"", > "Basic realm=\"security\" charset=\"UTF-8\"" > ] > } > > ``` 1. The clients have their dns name as :1-2-3-4.es. **pod**.cluster.local (which is also reflected in their certificate SubjectName. The clients are behind a load balancer whose dns name is es.es. **svc**.cluster.local (also reflected in the certificate SubjectName). Could this be the cause of missing auth token?) ---
**Author:** ![jbourne](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jbourne](https://discuss.elastic.co/u/jbourne)\ **Post date:** [February 13, 2019, 10:54pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/4 "2019-02-13T22:54:25Z")
Also, as a follow-up to my previous response. I changed the password for the elastic user using the elastic-setup-passwords script and tried using the new password in the curl request. It now gives me this error: _curl_ _-vvv -uelastic --cacert_ /usr/share/elasticsearch/config/ca.crt [https://ct-es.ct-es.svc.cluster.local:9200/](https://ct-es.ct-es.svc.cluster.local:9200/)_cluster/health?pretty_ ``` { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "failed to authenticate user [elastic]", "header" : { "WWW-Authenticate" : [ "Bearer realm=\"security\"", "Basic realm=\"security\" charset=\"UTF-8\"" ] } } ], "type" : "security_exception", "reason" : "failed to authenticate user [elastic]", "header" : { "WWW-Authenticate" : [ "Bearer realm=\"security\"", "Basic realm=\"security\" charset=\"UTF-8\"" ] } }, "status" : 401 } ``` ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [February 13, 2019, 11:30pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/5 "2019-02-13T23:30:56Z")
> [@jbourne](#): > > What's this in the HTTP header request- That's in the error response, not the request. Those are the authentication headers that are supported by your configuration of that Elasticsearch node. ```auto Bearer realm="security" ``` That's support for an OAuth2 Bearer access token ```auto Basic realm="security" charset="UTF-8" ``` That's support for Basic Authentication (username + password) If you fail to provide any authentication credentials the response will include HTTP response headers that are used to provide feedback to your user agent which HTTP authentication methods would be supported by the server. > [@jbourne](#): > > The clients have their dns name as :1-2-3-4.es. **pod**.cluster.local (which is also reflected in their certificate SubjectName. The clients are behind a load balancer whose dns name is es.es. **svc**.cluster.local (also reflected in the certificate SubjectName). Could this be the cause of missing auth token?) No. HTTP client authentication doesn't do reverse DNS lookups on the client address. The subject name in a client certificate is used for identity but it doesn't need to match any DNS names. > [@jbourne](#): > > I've looked at PKI realm documentation and it hasn't helped. Have you setup a PKI realm on your client node? It wasn't included in the `elasticsearch.yml` of your original post. > [@jbourne](#): > > I tried providing a client certificate and the key but still keep getting the same result. Please provide details. How did you do this? > [@jbourne](#): > > I changed the password for the elastic user using the elastic-setup-passwords script and tried using the new password in the curl request. It now gives me this error: At a guess it looks like you provided the wrong password, but there are other possible causes. The elasticsearch logs can help work out the cause. ---
**Author:** ![jbourne](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jbourne](https://discuss.elastic.co/u/jbourne)\ **Post date:** [February 13, 2019, 11:49pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/6 "2019-02-13T23:49:56Z")
I'm making this curl request from one of the client nodes. _curl_ _--cert /usr/share/elasticsearch/config/elasticsearch.crt_ _--key_ /usr/share/elasticsearch/config/elasticsearch.key --cacert /usr/share/elasticsearch/config/ca.crt [https://ct-es.ct-es.svc.cluster.local:9200/](https://ct-es.ct-es.svc.cluster.local:9200/)_cluster/health?pretty_ ``` { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "missing authentication token for REST request [/_cluster/health?pretty]", "header" : { "WWW-Authenticate" : [ "Bearer realm=\"security\"", "Basic realm=\"security\" charset=\"UTF-8\"" ] } } ], "type" : "security_exception", "reason" : "missing authentication token for REST request [/_cluster/health?pretty]", "header" : { "WWW-Authenticate" : [ "Bearer realm=\"security\"", "Basic realm=\"security\" charset=\"UTF-8\"" ] } }, "status" : 401 } ``` The PKI realm has been setup on both client and master nodes. If you take a closer look at my original post, you'll see I've mentioned that the elasticsearch.yml is from the client node. The master nodes have a similar looking YAML config with the exception that the node.master value is set to true. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [February 14, 2019, 1:16am UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/7 "2019-02-14T01:16:38Z")
> [@jbourne](#): > > The PKI realm has been setup on both client and master nodes. Sorry, that was my fault. The bottom of your file wasn't visible without scrolling and I missed it. It looks like you've missed enabling client authentication on the http interface. You need to set `xpack.security.http.ssl.client_authentication` to `optional` or `required`. You've set `xpack.ssl.client_authentication`, but the http interface does not rely on that default setting. ---
**Author:** ![jbourne](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jbourne](https://discuss.elastic.co/u/jbourne)\ **Post date:** [February 14, 2019, 5:38pm UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/8 "2019-02-14T17:38:48Z")
Thank you for the suggestion. So, I made that change you recommended and noticed something in the elasticsearch logs. Something about an empty certificate chain. I was able to remediate that by changing the xpack.security.http.ssl.certificate\_authorities setting to point to the location (instead of an array inside the [] parenthesis). Now, I'm getting this error: ``` { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "action [cluster:monitor/health] is unauthorized for user [1-2-3-4.ct-es.pod.cluster.local]" } ], "type" : "security_exception", "reason" : "action [cluster:monitor/health] is unauthorized for user [1-2-3-4.ct-es.pod.cluster.local]" }, "status" : 403 } ``` I understand this is related to role-mapping, however, I don't know how to work around that. These elasticsearch nodes are running on kubernetes pods which get their certificate dynamically during the bootstrapping process. The config files are mapped using Kubernetes ConfigMap. I'm not sure how to add the PKI\_DN for all the nodes in role\_mapping.yaml. Does the role mapping yaml file interpret a wildcard CN as shown below: role\_mapping.yml: client\_node: -"CN=\*.pod.cluster.local,O=My Organization, L=My City, ST=My State, C=My Country" ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [February 15, 2019, 7:32am UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/9 "2019-02-15T07:32:57Z")
Hi, Yes, you can use wildcards and lucene regular expressions in role mapping rules, see [https://www.elastic.co/guide/en/elasticsearch/reference/current/role-mapping-resources.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/role-mapping-resources.html) Alternatively if you ensure that only these clients will ever get a certificate that is signed by the CA that is configured as your CA for the http layer ```auto xpack.security.http.ssl.certificate_authorities: ``` you could possibly use the [role mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/security-api-put-role-mapping.html) to set a rule to match the realm name ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 15, 2019, 7:33am UTC](https://discuss.elastic.co/t/need-help-with-x-pack-tls-authentication/168121/10 "2019-03-15T07:33:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.