Elasticsearch security PKI

Hi everyone,

I'm trying to set up security in ES by following this guide here:

I've hit a roadblock and can't seem to work it out on my own

First issue is due to possibly breaking changes in latest elasticsearch (7.3.0)

Enabling PKI authentication from the guide doesn't work for me

xpack.security.authc.realms.pki1.type: pki

I've changed it to

xpack.security.authc.realms.pki.pki1.order: 1

So up until this point everything is working

Creating a client certificate command however fails

bin/elasticsearch-certutil cert --ca \
config/certs/elastic-stack-ca.p12 \
-name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
ENTER
client.p12 ENTER
ENTER

The -name basically ends up being the name of the certificate????

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 -name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

Enter password for CA (/etc/elasticsearch/certs/elastic-stack-ca.p12) : 
Please enter the desired output file [CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12]: 
Enter password for CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12 : 

Certificates written to /usr/share/elasticsearch/CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12

This file should be properly secured as it contains the private key for 
your instance.

This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
root@elastic18:/etc/elasticsearch/certs# ls
elastic-certificates.p12  elastic-stack-ca.p12
root@elastic18:/etc/elasticsearch/certs# ls /usr/share/elasticsearch/
 bin  'CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12'   jdk   lib   modules   NOTICE.txt   plugins   README.textile

I've tried using something like this instead:

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --name "CN=something,OU=Consulting Team,DC=mydomain,DC=com" --out /etc/elasticsearch/certs/client.p12

I generate the Private Key, Public Certificate and CA Certificate

openssl pkcs12 -in /etc/elasticsearch/certs/client.p12 -nocerts -nodes --out /etc/kibana/certs/client.key
openssl pkcs12 -in /etc/elasticsearch/certs/client.p12 -clcerts -nokeys --out /etc/kibana/certs/client.cer
openssl pkcs12 -in /etc/elasticsearch/certs/client.p12 -cacerts -nokeys -chain --out /etc/kibana/certs/client-ca.cer

I keep following the guide and add the following to kibana.yml

elasticsearch.hosts: ["https://localhost:9200"]
server.host: "localhost"
elasticsearch.username: "kibana"
elasticsearch.password: "pass"
xpack.security.enabled: true
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

I restart Kibana and receive no errors, but in the next step the curl command fails to authenticate using certificates only

curl https://localhost:9200/_xpack/security/_authenticate?pretty --key client.key --cert client.cer --cacert client-ca.cer -k -v

  • Trying ::1...
  • TCP_NODELAY set
  • Connected to localhost (::1) port 9200 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: client-ca.cer
    CApath: /etc/ssl/certs
  • (304) (OUT), TLS handshake, Client hello (1):
  • (304) (IN), TLS handshake, Server hello (2):
  • (304) (OUT), TLS change cipher, Client hello (1):
  • (304) (OUT), TLS handshake, Client hello (1):
  • (304) (IN), TLS handshake, Server hello (2):
  • (304) (IN), TLS Unknown, Certificate Status (22):
  • (304) (IN), TLS handshake, Unknown (8):
  • (304) (IN), TLS handshake, Request CERT (13):
  • (304) (IN), TLS handshake, Certificate (11):
  • (304) (IN), TLS handshake, CERT verify (15):
  • (304) (IN), TLS handshake, Finished (20):
  • (304) (OUT), TLS Unknown, Certificate Status (22):
  • (304) (OUT), TLS handshake, Certificate (11):
  • (304) (OUT), TLS Unknown, Certificate Status (22):
  • (304) (OUT), TLS handshake, CERT verify (15):
  • (304) (OUT), TLS Unknown, Certificate Status (22):
  • (304) (OUT), TLS handshake, Finished (20):
  • SSL connection using unknown / TLS_AES_256_GCM_SHA384
  • ALPN, server did not agree to a protocol
  • Server certificate:
  • subject: CN=instance
  • start date: Aug 19 01:53:44 2019 GMT
  • expire date: Aug 18 01:53:44 2022 GMT
  • issuer: CN=Elastic Certificate Tool Autogenerated CA
  • SSL certificate verify ok.
  • (304) (OUT), TLS Unknown, Unknown (23):

GET /_xpack/security/_authenticate?pretty HTTP/1.1
Host: localhost:9200
User-Agent: curl/7.58.0
Accept: /

  • (304) (IN), TLS Unknown, Certificate Status (22):
  • (304) (IN), TLS handshake, Newsession Ticket (4):
  • (304) (IN), TLS Unknown, Unknown (23):
    < HTTP/1.1 401 Unauthorized
    < WWW-Authenticate: Bearer realm="security"
    < WWW-Authenticate: ApiKey
    < WWW-Authenticate: Basic realm="security" charset="UTF-8"
    < content-type: application/json; charset=UTF-8
    < content-length: 746
    <
    {
    "error" : {
    "root_cause" : [
    {
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_xpack/security/_authenticate?pretty]",
    "header" : {
    "WWW-Authenticate" : [
    "Bearer realm="security"",
    "ApiKey",
    "Basic realm="security" charset="UTF-8""
    ]
    }
    }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_xpack/security/_authenticate?pretty]",
    "header" : {
    "WWW-Authenticate" : [
    "Bearer realm="security"",
    "ApiKey",
    "Basic realm="security" charset="UTF-8""
    ]
    }
    },
    "status" : 401
    }
    • Connection #0 to host localhost left intact

Thanks for your help

Correct, the blog post was written for 6.x , and 7.0 introduced this breaking change in the configuration settings format for realms.

As you can see in https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html:

--name <file_name>
    Specifies the name of the generated certificate. This parameter cannot be used with the ca parameter.

it also defaults to being the subject of the generated certificate.

This is expected as you haven't configured Kibana to connect to Elasticsearch with a certificate and key.

The error message could be better, and/or more instructive :(. The PKI realm is not available with the basic license ( see here under the Security column in basic. You need to start a trial license if you want to test this out, or look into getting one of the higher tier licenses available.

Thanks for the time to explain things to me Ioannis

I wasn't sure whether it was free or not, I take it basic security is free which allows the use of elastic username and password and PKI authentication requires a license.

I thought adding this to kibana.yml will allow me to use the certificate to authenticate

xpack.security.enabled: true
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

Since it's not free there's no point in pursuing it further. Thanks for the clarification

Correct, but not only the elastic user and the rest of the built-in user, but you can use the native realm and the file realm also and add your users, use RBAC etc. Read this blog post that has more information on what is available in basic

No, this is not enough. You'd need to also tell kibana which certificate and Key to use to authenticate to Elasticsearch with. The elasticsearch.ssl.certificateAuthorities only tells Kibana if it can trust the server certificate that Elaticsearch will present when kibana will attempt to connect to it. There is a section below in that blog post named "Using PKI to authenticate Kibana to the Elasticsearch cluster" which would describe how you'd go about it.

Understood ! FYI, we raised https://github.com/elastic/elasticsearch/issues/45728 to ensure that we will print a more informative error message that will notify the user of the cause of the authentication failure in similar cases.

Thank you again.
I couldn't find the section Using PKI to authenticate Kibana to the Elasticsearch cluster under the blog you linked.

I will try to set up everything from scratch again tomorrow, might leave out the file and native realm for now, and follow the guide you linked. Once that is working I can try and understand the native and file ream options.

Thanks for opening the issue on github

That section exists in the blog you linked, the one you were following through.

The blog I linked is just an overview of the security features available in basic.

We do have a nice tutorial to set up security in your cluster. Read about what security means for your cluster here : https://www.elastic.co/guide/en/elastic-stack-overview/7.3/elasticsearch-security.html and you can start with the Getting Started tutorial in https://www.elastic.co/guide/en/elastic-stack-overview/7.3/security-getting-started.html that goes through authentication and https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html that discusses communications encryption

Thanks for your help once more

I've finished setting up Elasticsearch part but stuck on encrypting communications in Kibana

I don't see any clear examples but trying to understand what you have written here

I'm still only using localhost and nginx to proxy 5601

So far nothing I tried so far has worked. I followed your suggestion from the link in the previous post

EXPORT CA Certificate

openssl pkcs12 -in /etc/elasticsearch/certs/elastic-stack-ca.p12 -clcerts -nokeys -chain -out /etc/kibana/certs/ca.pem

EXPORT Cert and Key for Kibana

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --dns elastic18

kibana.yml

elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "elasticsearch.password"

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/instance.crt
server.ssl.key: /etc/kibana/certs/instance.key


elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca.pem
elasticsearch.ssl.verificationMode: certificate

Error from Kibana

Aug 21 12:53:31 elastic18 kibana[58413]: {"type":"error","@timestamp":"2019-08-21T02:53:31Z","tags":["connection","client","error"],"pid":58413,"level":"error","error":{"message":"140001888880448:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n","name":"Error","stack":"Error: 140001888880448:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"},"message":"140001888880448:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"}

If I comment out the # server.ssl.enabled: true
It's working but that means ssl is not enabled doesn't it?

Thanks for all your help

EDIT:

Changed the nginx proxy_pass directive from http to https and seems okay now

location / {
proxy_pass https://localhost:5601;

Trying to get the logstash part going but can't get rid of this error

Manticore::UnknownException: Host name 'localhost' does not match the certificate subject provided by the peer (CN=instance)>

I've read one of your posts and you talk about making sure to set the --dns properly which I thought I was using correctly

I've tried to set it to localhost, then I tried instance then I tried ips as well

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --dns instance,localhost,elastic18 --name instance --ip 127.0.0.1,192.168.131.140 --out /etc/kibana/certs/kibana.zip

openssl x509 -noout -checkhost localhost -in instance.crt 
Hostname localhost does match certificate
openssl x509 -noout -checkhost instance -in instance.crt 
Hostname instance does match certificate
openssl x509 -noout -checkhost elastic18 -in instance.crt 
Hostname elastic18 does match certificate

Any suggestions Ioannis?
I appreciate the help

i've been looking at this guide as well, to understand the process

This is the node name in logstash.yml

node.name: localhost

These are the certificates that you generate for Kibana. Logstash connects to Elasticsearch, not Kibana , and as such you need to make sure that the certificate that Elasticsearch uses for TLS has the right SANs.

The certificate that Logstash has an issue with, is the certificate that is defined by

xpack.security.http.ssl.keystore.path

or

xpack.security.http.ssl.certificate

depending on how you have configured Elasticsearch

1 Like

Thank you so much!!!

Like you said

openssl pkcs12 -nokeys -info -in elastic-certificates.p12 


Certificate bag
Bag Attributes
    friendlyName: instance
    localKeyID: 54 69 6D 65 20 31 35 36 36 31 37 39 36 32 38 37 32 32 
subject=CN = instance

issuer=CN = Elastic Certificate Tool Autogenerated CA

I recreated the certificate

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --name localhost --dns localhost --ip 127.0.0.1 --out /etc/elasticsearch/certs/elastic-certificates.p12

I am getting this warning, but not sure if it's false

2019-08-22T18:57:45,941][WARN ][logstash.outputs.elasticsearch] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true

Maybe it applies to this

If you have set

:ssl_certificate_verification => false

in your Logstash's Elasticsearch output plugin configuration then you need to remove that line. If not, this is probably the bug you have referenced and you can feel free to ignore

I haven't set it, and I've also tried setting it to true, didn't make a difference. Warning was still present.
I'll ignore it and keep going with the rest of the setup.

You've helped me immensely, once again thank you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.