Need help with X-pack TLS authentication

Elastic Stack Elasticsearch
1

I'm trying to setup an elastic stack on kubernetes in AWS. I have:

1 - 3 master nodes
2 - 2 client nodes
3 - 1 data node

In Kubernetes, I've created a Service that exposes ports 9200 and 9300. This service gets applied to an AWS ELB which creates listeners for those ports. The full DNS name for the ELB is ct-es.ct-es.svc.clsuter.local. The pods for their respective containers execute successfully but X-pack and PKI authentication is where I'm running into problems.

When I try to execute a curl request shown below, I get a 401 error in the response:

_curl -X GET https://ct-es.ct-es.svc.cluster.local:9200/cluster/health?pretty --cacert /usr/share/elasticsearch/config/ca.crt

{
         "error" : {
            "root_cause" : [
              {
                "type" : "security_exception",
                "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
                "header" : {
                  "WWW-Authenticate" : [
                    "Bearer realm=\"security\"",
                    "Basic realm=\"security\" charset=\"UTF-8\""
                  ]
                }
              }
            ],
            "type" : "security_exception",
            "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
            "header" : {
              "WWW-Authenticate" : [
                "Bearer realm=\"security\"",
                "Basic realm=\"security\" charset=\"UTF-8\""
              ]
            }
          },
          "status" : 401
        }
}

This is what my elasticsearch.yml file looks like on the client node:

cluster:
  name: logs

xpack.license.self_generated.type: trial
xpack.monitoring.enabled: true
xpack.security.enabled: true
xpack.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
xpack.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
xpack.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/ca.crt" ]
xpack.ssl.client_authentication: required
xpack.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/ca.crt" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key 
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/ca.crt" ]
xpack.security.audit.enabled: true
xpack.security.audit.outputs: [ index, logfile ]
xpack.security.audit.index.settings:
  index:
    number_of_shards: 1
    number_of_replicas: 1
xpack.security.authc.realms:
  realm1:
    type: native
    order: 0
  realm2:
    type: pki
    order: 1

network.host: _eth0_

thread_pool.bulk.queue_size: 800

path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs

http:
  enabled: true
  compression: true
2

You have security enabled in Elasticsearch with 2 security realms (native and PKI) so you need to send some client credentials in your request. You need to send either

  • a username and password combination for a user that exists in the native realm , with -uusername:password

  • a client certificate by specifying --cert and --key , see the curl man page at curl - How To Use

Please also read through our PKI realm documentation, it will be really helpful.

3

I've looked at PKI realm documentation and it hasn't helped. I'd like authentication to happen using PKI because that removes the hassle of maintaining passwords.
I tried providing a client certificate and the key but still keep getting the same result. My questions are:

  1. What's this in the HTTP header request-
"header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  1. The clients have their dns name as :1-2-3-4.es.pod.cluster.local (which is also reflected in their certificate SubjectName. The clients are behind a load balancer whose dns name is es.es.svc.cluster.local (also reflected in the certificate SubjectName). Could this be the cause of missing auth token?)
4

Also, as a follow-up to my previous response. I changed the password for the elastic user using the elastic-setup-passwords script and tried using the new password in the curl request. It now gives me this error:

curl -vvv -uelastic --cacert /usr/share/elasticsearch/config/ca.crt https://ct-es.ct-es.svc.cluster.local:9200/cluster/health?pretty

{
      "error" : {
        "root_cause" : [
          {
            "type" : "security_exception",
            "reason" : "failed to authenticate user [elastic]",
            "header" : {
              "WWW-Authenticate" : [
                "Bearer realm=\"security\"",
                "Basic realm=\"security\" charset=\"UTF-8\""
              ]
            }
          }
        ],
        "type" : "security_exception",
        "reason" : "failed to authenticate user [elastic]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      },
      "status" : 401
}
5

That's in the error response, not the request.
Those are the authentication headers that are supported by your configuration of that Elasticsearch node.

Bearer realm="security"

That's support for an OAuth2 Bearer access token

Basic realm="security" charset="UTF-8"

That's support for Basic Authentication (username + password)

If you fail to provide any authentication credentials the response will include HTTP response headers that are used to provide feedback to your user agent which HTTP authentication methods would be supported by the server.

No.
HTTP client authentication doesn't do reverse DNS lookups on the client address. The subject name in a client certificate is used for identity but it doesn't need to match any DNS names.

Have you setup a PKI realm on your client node?
It wasn't included in the elasticsearch.yml of your original post.

Please provide details. How did you do this?

At a guess it looks like you provided the wrong password, but there are other possible causes. The elasticsearch logs can help work out the cause.

6

I'm making this curl request from one of the client nodes.
curl --cert /usr/share/elasticsearch/config/elasticsearch.crt --key /usr/share/elasticsearch/config/elasticsearch.key --cacert /usr/share/elasticsearch/config/ca.crt https://ct-es.ct-es.svc.cluster.local:9200/cluster/health?pretty

    {
      "error" : {
        "root_cause" : [
          {
            "type" : "security_exception",
            "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
            "header" : {
              "WWW-Authenticate" : [
                "Bearer realm=\"security\"",
                "Basic realm=\"security\" charset=\"UTF-8\""
              ]
            }
          }
        ],
        "type" : "security_exception",
        "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      },
      "status" : 401
    }

The PKI realm has been setup on both client and master nodes. If you take a closer look at my original post, you'll see I've mentioned that the elasticsearch.yml is from the client node. The master nodes have a similar looking YAML config with the exception that the node.master value is set to true.

7

Sorry, that was my fault. The bottom of your file wasn't visible without scrolling and I missed it.

It looks like you've missed enabling client authentication on the http interface. You need to set xpack.security.http.ssl.client_authentication to optional or required. You've set xpack.ssl.client_authentication, but the http interface does not rely on that default setting.

8

Thank you for the suggestion. So, I made that change you recommended and noticed something in the elasticsearch logs. Something about an empty certificate chain. I was able to remediate that by changing the xpack.security.http.ssl.certificate_authorities setting to point to the location (instead of an array inside the [ ] parenthesis).
Now, I'm getting this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [cluster:monitor/health] is unauthorized for user [1-2-3-4.ct-es.pod.cluster.local]"
      }
    ],
    "type" : "security_exception",
    "reason" : "action [cluster:monitor/health] is unauthorized for user [1-2-3-4.ct-es.pod.cluster.local]"
  },
  "status" : 403
}

I understand this is related to role-mapping, however, I don't know how to work around that. These elasticsearch nodes are running on kubernetes pods which get their certificate dynamically during the bootstrapping process. The config files are mapped using Kubernetes ConfigMap. I'm not sure how to add the PKI_DN for all the nodes in role_mapping.yaml. Does the role mapping yaml file interpret a wildcard CN as shown below:

role_mapping.yml:

client_node:
-"CN=*.pod.cluster.local,O=My Organization, L=My City, ST=My State, C=My Country"

9

Hi,

Yes, you can use wildcards and lucene regular expressions in role mapping rules, see https://www.elastic.co/guide/en/elasticsearch/reference/current/role-mapping-resources.html

Alternatively if you ensure that only these clients will ever get a certificate that is signed by the CA that is configured as your CA for the http layer

xpack.security.http.ssl.certificate_authorities:

you could possibly use the role mapping API to set a rule to match the realm name

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.