Configure PKI authentication for Elasticsearch 7.7

This guide Elasticsearch Security: Configure TLS/SSL & PKI Authentication | Elastic Blog shows how to configure PKI for Elasticsearch version 6.5 however this doesn't work for version 7.7.0 ( the version I'm currently using ). When I add xpack.security.authc.realms.pki1.type: pki to my elasticsearch.yml file, I get the error below:

java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.security.Security] Likely root cause: java.lang.IllegalArgumentException: Incorrect realm settings found. Realm settings have been changed to include the type as part of the setting key. For example 'xpack.security.authc.realms.file.my_file.order' Found invalid config: xpack.security.authc.realms.pki1.type Please see the breaking changes documentation.

I went through the documentation for Elasticsearch 7.7 but didn't find anything specific to PKI changes. Can someone please assist with the settings to use and also how to generate the client certificate for authentication? Thanks

Elasticsearch 7.7 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

Does this help?

Add a realm configuration for a pki realm to elasticsearch.yml under the xpack.security.authc.realms.pki namespace.

Thanks for the reply. Using the settings below works. That is Elasticsearch now starts up without an error. Would you know how I can configure the client and server certificates to use for mutual authentication?

xpack:
  security:
    authc:
      realms:
        pki:
          pki1:
            order: 1

After further research, I was able to use the certutil command to generate a client cert. Elasticsearch however still seems to require a username and password.

Command

bin/elasticsearch-certutil cert --ca-cert certs/ca/ca.crt --ca-key certs/ca/ca.key -out client.p12

I then used this client.p12 cert to authenticate to the cluster

curl -k --cert-type P12 --cert client.p.12 https://localhost:9200

I get a 401 error

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}%

I'm able to authenticate when I provide the username and password

 curl -k --cert-type P12 --cert client.p.12 https://localhost:9200  -u user:password
{
  "name" : "node01",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "W0MJXDsnTw2LlnhGHw3Lug",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-12T02:01:37.602180Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

How do I connect to Elasticsearch with the certificate only and NOT provide a username and password? My elasticsearch.yml is below

node.name: node01
xpack.security.enabled: true
discovery.type: single-node
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: certs/node01.key
xpack.security.transport.ssl.certificate: certs/node01.crt
xpack.security.transport.ssl.certificate_authorities: [ "certs/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node01.key
xpack.security.http.ssl.certificate: certs/node01.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.http.ssl.client_authentication: optional

Thanks for your help.

I've still not found a solution to Elasticsearch still requiring basic auth despite trying to use a certificate to authenticate. What setting(s) do I have to add to elasticsearch.yml to make cert authentication work?

What is the complete content of your elasticsearch.yml file? The one you shared immediately above does not include the PKI realm settings.

PKI realm is a license feature. So you'll need either a platinum license or trial license. Otherwise, the PKI realm will be skipped.

This is the entire elasticsearch.yml file

node.name: node01
xpack.security.enabled: true
discovery.type: single-node
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: certs/node01.key
xpack.security.transport.ssl.certificate: certs/node01.crt
xpack.security.transport.ssl.certificate_authorities: [ "certs/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node01.key
xpack.security.http.ssl.certificate: certs/node01.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.http.ssl.client_authentication: required
xpack.security.http.ssl.verification_mode: certificate



xpack:
  security:
    authc:
      realms:
        pki:
          pki1:
            order: 1
            certificate_authorities: [ "certs/ca.crt" ]

when I try to connect without the -k flag I get the error below

curl --cert-type P12 --cert client_node.p12 https://localhost:9200

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

and in verbose mode

curl --cert-type P12 --cert client_node.p12 https://localhost:9200 -vvv
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

which certificate is curl trying to verify and is the command I'm using the right way to connect using SSL authentication?