Verify PKI enabling in elastic nodes

Elastic Stack Elasticsearch
1

I have enabled the PKI in elasticsearch.yml as below:
xpack.security.authc.realms.pki1.type: pki
xpack.security.authc.realms.pki1.order: 1
xpack.security.authc.realms.pki1.certificate_authorities: [ path to cacert ]

Where the first realm is the native realm of order 0.

How to verify if PKI is enabled in my elastic nodes.

2

Hi,

Have you gone through our documentation ? It's very detailed, I'm sure it will help you out !

After you verify that you have done all is required (such as enabling TLS for the http layer and setting client_configuration to either required or optional ,among others) you can try with curl for instance by passing a client certificate and key : i.e.

curl -k -v --cert path/to/your/client/certificate.pem --key path/to/your/client/key.pem https://localhost:9200

Mind you the client certificate needs to be signed by the CA cert that you have configured as xpack.security.authc.realms.pki1.certificate_authorities

3

I have enabled TLS for the http layer and also set http client_configuration to optional.

And i tried your curl command i am getting the below error:

  • About to connect() to server @ port ####
  • Trying server...
  • Connected to server (IP) port ####
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
  • NSS error -8178 (SEC_ERROR_BAD_KEY)
  • Peer's public key is invalid.
  • Closing connection 0
    curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

My client certificate is signed by the CA cert configured in : xpack.security.authc.realms.pki1.certificate_authorities

I have also created the cacert,cert and private key using the command:
./certutil cert --ip --dns --name --pem -v.

Please let me know where i am going wrong. Is the certificate creation method or wrong PKI settings.

4

See point 4 in the doc I shared above.

Your http layer TLS configuration need to also trust the client certificate. So you also need to set

xpack.security.http.ssl.certificate_authorities

with the same value as

xpack.security.authc.realms.pki1.certificate_authorities

or

xpack.security.http.ssl.verification_mode: none

( I wouldn't suggest that)

1 Like
5

Thanks for the reply.
To test the PKI settings i used the below command:
curl -k -v --cert path/to/your/client/certificate.pem --key path/to/your/client/key.pem https://localhost:9200

But the above command misses the Username and password. If i give the elastic username and password i am able to hit https://localhost:9200 through curl command.

But what is the use of enabling PKI.
As per my knowledge,
I am trying to use mutual authentication TLS and PKI authentication. So that only authorized clients can connect to the HTTP port, and their certificate provides the totality of their credentials, so they do not need to provide a username and password to authenticate their ES requests on the REST interface.
But even after i created application specific certificate using specific dns and tried to authenticate the user , it still prompts for user password.

Please correct me if i am wrong.

6

Hi @B123,

I am working with error that you posted earlier:

Seems like the curl was not able to load the client key to present to ES server.
This might be because of may be missing passphrase or different format key.
Or this could be due to different format:

Could you please check and see if anything above helps you resolve the problem?
If not please provide with the command with the verbose output so we can take a look.

Thanks and Regards,
Yogesh Gaikwad

7

Hi,
I have used the verbose mode for my command as mentioned by @ikakavas :

Actually i am not sure if my certificate creation method is correct because it dint prompt for any password .
I used the below command for cert, cacert and private key generation :

./certutil cert --ip --dns --name --pem -v

8

This can't be correct. All --ip and --dns and --name need a value after them, they're not boolean options like i.e. --pem. I am assuming that you are using 6.2 since your cli tool is named certutil and not elasticsearch-certutil so please read through our documentation on how to use the tool to create the certificate and keys that you need

9

Yes i am using 6.2 version of elasticsearch. Therefore using certutil and I have passes values for --ip and --dns and --name.

./certutil cert --ip XX.XX.XX.XXX --dns "CN=####,OU=XXX,DC=XXX" --name "cert_for_prac" --pem -v

But the above command did not prompt for any password. Like Enter password for CA:

And while using the curl command

getting below error:

  • About to connect() to XXXX port #### (#0)
  • Trying (IP OF SERVER)...
  • Connected to XXXX (IP OF SERVER) port ####(#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • NSS: client certificate from file
  •   subject: CN=cert_for_prac

  •   start date: Jan 31 13:49:24 2019 GMT

  •   expire date: Jan 30 13:49:24 2022 GMT

  •   common name: cert_for_prac

  •   issuer: CN=Elastic Certificate Tool Autogenerated CA
  • NSS error -12224 (SSL_ERROR_CERTIFICATE_UNKNOWN_ALERT)
  • SSL peer had some unspecified issue with the certificate it received.
  • Closing connection 0
    curl: (35) SSL peer had some unspecified issue with the certificate it received.
10

Now this is a different error than the one you shared above, so you must have changed something.

It now says:

SSL peer had some unspecified issue with the certificate it received.
Closing connection 0
curl: (35) SSL peer had some unspecified issue with the certificate it received.

so that leads me to believe that either you don't pass the right files for --cert and --key OR you still haven't applied the changes I suggested in a previous response . Could you please make sure ? You could share your whole elasticsearch.yml (scrubbing off sensitive data of course )

11

Hi,

Yes i did. I was previously using a different cert which i changed later since i was getting error. Still facing SSL issue.

Find below my elasticsearch.yml xpack config:

xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.client_authentication: optional

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: private.key
xpack.security.transport.ssl.certificate:
xpack.security.transport.ssl.certificate_authorities:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: private.key
xpack.security.http.ssl.certificate:
xpack.security.http.ssl.certificate_authorities:

#authentication for elastic nodes
xpack.security.authc.realms.pki1.type: pki
xpack.security.authc.realms.pki1.order: 0
xpack.security.authc.realms.pki1.certificate_authorities:

xpack.security.authc.realms.native1.type: native
xpack.security.authc.realms.native1.order: 1

NOTE: I am using different set of certificates for ssl enabling(already configured for my cluster) and for PKI (generated using certutil command) .
How do i combine these certificates (import if possible into a single certificate).
Or please suggest a best way @ikakavas

12

In my answer from yesterday I mentioned that this is not what you should be doing, as we already also document in our PKI realm documentation that I've linked above.

As I mentioned yesterday, you can either

  1. Use the same value for xpack.security.http.ssl.certificate_authorities: and xpack.security.authc.realms.pki1.certificate_authorities:

From what you are saying, this is an existing PEM file that you already have, and you should already have the corresponding key file from when you generated it. Let's assume ca.crt and ca.key accordingly.
In order to generate new key and certificate pairs that are signed by this CA, you need to run
./certutil --ca-cert ca.crt --ca-key ca.key --ip XX.XX.XX.XXX --dns "CN=####,OU=XXX,DC=XXX" --name "cert_for_prac" --pem -v
and use that when you run curl.

  1. Set
    xpack.security.http.ssl.verification_mode: none
    and keep using the certificates you have already generated.

I suggest you go with option 1.

13

Thanks a lot for the reply @ikakavas.

I used your suggestion mentioned in Option 1 and ran the certutil command:
./certutil --ca-cert ca.crt --ca-key ca.key --ip XX.XX.XX.XXX --dns "CN=####,OU=XXX,DC=XXX" --name "cert_for_prac" --pem -v

I am not able to figure out which CN ie user to give here. And any specific user i give here where will the password of the user be stored so that he can be authenticated in the PKI realm.
Currently i am using only two realms. Where the native realm authenticates elastic,kibana and logstash inline users on providing their respective passwords (as per my knowledge).
Please correct me if i am wrong.
And Please let me know how the users must be configured for PKI realm.

Currently while executing the below curl command:
curl -k -v "https://localhost:9200/_xpack/security/_authenticate?pretty" --cacert 'ca.crt' --key 'ca.key'

I am Getting the below output :
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< content-type: application/json; charset=UTF-8
< content-length: 572
<
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

14

Currently while executing the below curl command:
curl -k -v "https://localhost:9200/_xpack/security/_authenticate?pretty" --cacert 'ca.crt' --key 'ca.key'

You should not be using ca.crt and ca.key for authentication. You should be using ca.crt and ca.key to generate a new certificate and key with ./certutil - as I described above - . If you use
--name "cert_for_prac" for example , the files will be named cert_for_prac.crt and cert_for_prac.key.

Then you can use cert_for_prac.crt and cert_for_prac.key with curl as in

curl -k -v --cert cert_for_prac.crt --key cert_for_prac.key

There is no rule by Elasticsearch as to what is the DN of the users in your PKI realm

PKI realm users authenticate by providing their client key and certificate. They don't have a password. If a user has a private key and a certificate that is trusted by the CA that you configure forcertificate_authorities then they can authenticate to Elasticsearch.
As we mention in the documentation that I've shared above, the CN of the user in the certificate gets mapped to a username in Elasticsearch. This doesn't mean that a user with that username is "created" in Elasticsearch (or in any other realm) but just that when a user with that private key/ certificate authenticates to Elasticsearch, they will be known as that username. If you don't want to use the CN but something else from the information in the certificate, the instructions are in the same document above.

Now when your user authenticates via the PKI realm, they have no permission to do anything as they are not assigned any roles by default. You need to decide and define the which roles they get by using the role mapping api (described in the same documentation)

Users elastic, kibana, etc are not in the native realm. They are buiilt-in users of the reserved realm. These users can only authenticate with username/password.

A native realm also uses username/password for authentication. You can create users to use in the native realm with the User Management APIs.

15

Hi,
While using

I am getting below error:

  • About to connect() to localhost port 9200 (#0)
  • Trying ...
  • Connected to localhost(IP) port 9200 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • NSS: client certificate from file
  •   subject: CN=cert_p

  •   start date: Feb 05 13:15:32 2019 GMT

  •   expire date: Feb 04 13:15:32 2022 GMT

  •   common name: cert_p

  •   issuer: CN=
  • NSS error -12224 (SSL_ERROR_CERTIFICATE_UNKNOWN_ALERT)
  • SSL peer had some unspecified issue with the certificate it received.
  • Closing connection 0
    curl: (35) SSL peer had some unspecified issue with the certificate it received.

But when i used below command

There are no certificate related issues. But still getting security exception.
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

The DN i have used while creating the certificate , I have used the same DN in my role mapping too. But still getting security exception. Should i add my DN to the users.
Please help me out @ikakavas.

16

means that Elasticsearch cannot validate the certificate you are sending.

Yes, because you don't actually send a client certificate for Elasticsearch to not validate.

Please see curl - How To Use fo an explanation of what the curl parameters actually do.

You need to share you Elasticsearch logs for us to see what error it was thrown, otherwise we can only guess.

17

@ikakavas elastic logs are below

18

This sound like something is obviously very wrong with the client certificate you used for curl . Can you please

  • Copy paste here the exact command with certutil that you used to create the cert_for_prac.key and cert_for_prac.crt
  • Copy paste here the exact curl command you are running
  • Copy paste here the output of the following two commans with openssl
    openssl rsa -in cert_for_prac.key -check -noout and
    openssl x509 -in cert_for_prac.crt -text -noout
19

Hi,

@ikakavas refer below:

  • ./certutil cert --ca-cert existing_cacert --ca-key eixting_private.key --ip ##.##.## --dns "CN=XXXX,OU=Standard Users,DC=value" --name "cert_for_prac" --pem -v
    where ( existing_cacert ie the one used for certificate_authorities under ssl setting and eixting_private.key ie the one used for key under setting in elasticsearch.yml)

  • curl -k -v "https://localhost:port//_cluster/health?pretty" --cert cert_for_prac.crt --key cert_for_prac.key

  • openssl rsa -in cert_for_prac.key -check -noout
    output: RSA key ok

openssl x509 -in cert_for_prac.crt -text -noout
output:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
Signature Algorithm: sha256WithRSAEncryption
Issuer: Name
Validity
Not Before: Feb 5 13:15:32 2019 GMT
Not After : Feb 4 13:15:32 2022 GMT
Subject: CN=cert_p
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
Values
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:

        X509v3 Authority Key Identifier:
            keyid:

        X509v3 Subject Alternative Name:
            DNS: One specified during certificate creation
        X509v3 Basic Constraints:
            CA:FALSE
Signature Algorithm: sha256WithRSAEncryption
20

These are not the same thing.

"eixting_private.key ie the one used for key under setting in elasticsearch.yml", i.e the key you use for xpack.security.http.ssl.key is an RSA key that corresponds to the certificate that you use for xpack.security.http.ssl.cert .

This is NOT the RSA key that corresponds to your CA cert, the one that you use for xpack.security.http.ssl.certificate_authorities

In other words eixting_private.key is _not ca.key I meant above. You need to have the CA private key in order to generate and sign new certificates by that CA. If you don't have that then you can't use this CA to generate new certificates and keys for your clients and you need to generate a new CA key pair with certutil CA (see our docs) . Then you would need to regenerate certificates and keys for elasticsearch and change all your settigns in elasticsearch.yml Finally, you'd use this CA to generate client keys and certificates for your clients to use for authentication in the PKI realm.