# Elasticsearch security PKI

**URL:** <https://discuss.elastic.co/t/elasticsearch-security-pki/195838>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [August 20, 2019, 4:02am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838 "2019-08-20T04:02:45Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 20, 2019, 4:02am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/1 "2019-08-20T04:02:45Z")

</div>

Hi everyone,

I'm trying to set up security in ES by following this guide here:

> **[Elasticsearch Security: Configure TLS/SSL & PKI Authentication](https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication)**
>
> Learn how to enable Elasticsearch security, configure TLS/SSL, use PKI for authentication, authenticate Kibana to an Elasticsearch cluster using PKI, and set passwords for built-in users.

I've hit a roadblock and can't seem to work it out on my own

First issue is due to possibly breaking changes in latest elasticsearch (7.3.0)

Enabling PKI authentication from the guide doesn't work for me

> `xpack.security.authc.realms.pki1.type: pki`

I've changed it to

> `xpack.security.authc.realms.pki.pki1.order: 1`

So up until this point everything is working

Creating a client certificate command however fails

```
bin/elasticsearch-certutil cert --ca \
config/certs/elastic-stack-ca.p12 \
-name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
ENTER
client.p12 ENTER
ENTER

```

The -name basically ends up being the name of the certificate????

```
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 -name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

Enter password for CA (/etc/elasticsearch/certs/elastic-stack-ca.p12) : 
Please enter the desired output file [CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12]: 
Enter password for CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12 : 

Certificates written to /usr/share/elasticsearch/CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12

This file should be properly secured as it contains the private key for 
your instance.

This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
root@elastic18:/etc/elasticsearch/certs# ls
elastic-certificates.p12 elastic-stack-ca.p12
root@elastic18:/etc/elasticsearch/certs# ls /usr/share/elasticsearch/
 bin 'CN=something,OU=Consulting Team,DC=mydomain,DC=com.p12' jdk lib modules NOTICE.txt plugins README.textile

```

I've tried using something like this instead:

> /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --name "CN=something,OU=Consulting Team,DC=mydomain,DC=com" --out /etc/elasticsearch/certs/client.p12

I generate the Private Key, Public Certificate and CA Certificate

```
openssl pkcs12 -in /etc/elasticsearch/certs/client.p12 -nocerts -nodes --out /etc/kibana/certs/client.key
openssl pkcs12 -in /etc/elasticsearch/certs/client.p12 -clcerts -nokeys --out /etc/kibana/certs/client.cer
openssl pkcs12 -in /etc/elasticsearch/certs/client.p12 -cacerts -nokeys -chain --out /etc/kibana/certs/client-ca.cer

```

I keep following the guide and add the following to kibana.yml

```
elasticsearch.hosts: ["https://localhost:9200"]
server.host: "localhost"
elasticsearch.username: "kibana"
elasticsearch.password: "pass"
xpack.security.enabled: true
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

```

I restart Kibana and receive no errors, but in the next step the curl command fails to authenticate using certificates only

> curl [https://localhost:9200/\_xpack/security/\_authenticate?pretty](https://localhost:9200/_xpack/security/_authenticate?pretty) --key client.key --cert client.cer --cacert client-ca.cer -k -v

- Trying ::1...
- TCP\_NODELAY set
- Connected to localhost (::1) port 9200 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: client-ca.cer  
CApath: /etc/ssl/certs
- (304) (OUT), TLS handshake, Client hello (1):
- (304) (IN), TLS handshake, Server hello (2):
- (304) (OUT), TLS change cipher, Client hello (1):
- (304) (OUT), TLS handshake, Client hello (1):
- (304) (IN), TLS handshake, Server hello (2):
- (304) (IN), TLS Unknown, Certificate Status (22):
- (304) (IN), TLS handshake, Unknown (8):
- (304) (IN), TLS handshake, Request CERT (13):
- (304) (IN), TLS handshake, Certificate (11):
- (304) (IN), TLS handshake, CERT verify (15):
- (304) (IN), TLS handshake, Finished (20):
- (304) (OUT), TLS Unknown, Certificate Status (22):
- (304) (OUT), TLS handshake, Certificate (11):
- (304) (OUT), TLS Unknown, Certificate Status (22):
- (304) (OUT), TLS handshake, CERT verify (15):
- (304) (OUT), TLS Unknown, Certificate Status (22):
- (304) (OUT), TLS handshake, Finished (20):
- SSL connection using unknown / TLS\_AES\_256\_GCM\_SHA384
- ALPN, server did not agree to a protocol
- Server certificate:
- subject: CN=instance
- start date: Aug 19 01:53:44 2019 GMT
- expire date: Aug 18 01:53:44 2022 GMT
- issuer: CN=Elastic Certificate Tool Autogenerated CA
- SSL certificate verify ok.
- (304) (OUT), TLS Unknown, Unknown (23):

> GET /\_xpack/security/\_authenticate?pretty HTTP/1.1  
> Host: localhost:9200  
> User-Agent: curl/7.58.0  
> Accept: _/_

- (304) (IN), TLS Unknown, Certificate Status (22):
- (304) (IN), TLS handshake, Newsession Ticket (4):
- (304) (IN), TLS Unknown, Unknown (23):  
\< HTTP/1.1 401 Unauthorized  
\< WWW-Authenticate: Bearer realm="security"  
\< WWW-Authenticate: ApiKey  
\< WWW-Authenticate: Basic realm="security" charset="UTF-8"  
\< content-type: application/json; charset=UTF-8  
\< content-length: 746  
\<  
{  
"error" : {  
"root\_cause" : [  
{  
"type" : "security\_exception",  
"reason" : "missing authentication credentials for REST request [/\_xpack/security/\_authenticate?pretty]",  
"header" : {  
"WWW-Authenticate" : [  
"Bearer realm="security"",  
"ApiKey",  
"Basic realm="security" charset="UTF-8""  
]  
}  
}  
],  
"type" : "security\_exception",  
"reason" : "missing authentication credentials for REST request [/\_xpack/security/\_authenticate?pretty]",  
"header" : {  
"WWW-Authenticate" : [  
"Bearer realm="security"",  
"ApiKey",  
"Basic realm="security" charset="UTF-8""  
]  
}  
},  
"status" : 401  
}
  - Connection #0 to host localhost left intact

Thanks for your help

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [August 20, 2019, 10:44am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/2 "2019-08-20T10:44:42Z")

</div>

> [@VamPikmin](#):
>
> First issue is due to possibly breaking changes in latest elasticsearch (7.3.0)

Correct, the blog post was written for 6.x , and 7.0 introduced this breaking change in the configuration settings format for realms.

> [@VamPikmin](#):
>
> The -name basically ends up being the name of the certificate????

As you can see in [https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html:](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html:)

```auto
--name <file_name>
    Specifies the name of the generated certificate. This parameter cannot be used with the ca parameter.

```

it also defaults to being the subject of the generated certificate.

> [@VamPikmin](#):
>
> I restart Kibana and receive no errors,

This is expected as you haven't configured Kibana to connect to Elasticsearch with a certificate and key.

> [@VamPikmin](#):
>
> but in the next step the curl command fails to authenticate using certificates only

The error message could be better, and/or more instructive :(. The PKI realm **is not** available with the basic license ( see [here](https://www.elastic.co/subscriptions) under the `Security` column in basic. You need to start a trial license if you want to test this out, or look into getting one of the higher tier licenses available.

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 20, 2019, 11:37am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/3 "2019-08-20T11:37:37Z")

</div>

Thanks for the time to explain things to me Ioannis

I wasn't sure whether it was free or not, I take it basic security is free which allows the use of elastic username and password and PKI authentication requires a license.

I thought adding this to kibana.yml will allow me to use the certificate to authenticate

> xpack.security.enabled: true  
> elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/client-ca.cer  
> elasticsearch.ssl.verificationMode: certificate

Since it's not free there's no point in pursuing it further. Thanks for the clarification

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [August 20, 2019, 11:49am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/4 "2019-08-20T11:49:08Z")

</div>

> [@VamPikmin](#):
>
> I take it basic security is free which allows the use of elastic username and password

Correct, but not only the `elastic` user and the rest of the built-in user, but you can use the [native realm](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-native-realm.html) and the [file realm](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-file-realm.html) also and add your users, use RBAC etc. Read this [blog post](https://www.elastic.co/blog/security-for-elasticsearch-is-now-free) that has more information on what is available in basic

> [@VamPikmin](#):
>
> I thought adding this to kibana.yml will allow me to use the certificate to authenticate

No, this is not enough. You'd need to also tell kibana which certificate and Key to use to authenticate to Elasticsearch with. The `elasticsearch.ssl.certificateAuthorities` only tells Kibana if it can trust the server certificate that Elaticsearch will present when kibana will attempt to connect to it. There is a section below in that blog post named "Using PKI to authenticate Kibana to the Elasticsearch cluster" which would describe how you'd go about it.

> [@VamPikmin](#):
>
> Since it's not free there's no point in pursuing it further. Thanks for the clarification

Understood ! FYI, we raised [Enhance logging when configured realms are not available in current license · Issue #45728 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/45728) to ensure that we will print a more informative error message that will notify the user of the cause of the authentication failure in similar cases.

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 20, 2019, 12:44pm UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/5 "2019-08-20T12:44:34Z")

</div>

Thank you again.  
I couldn't find the section `Using PKI to authenticate Kibana to the Elasticsearch cluster` under the blog you linked.

I will try to set up everything from scratch again tomorrow, might leave out the file and native realm for now, and follow the guide you linked. Once that is working I can try and understand the native and file ream options.

Thanks for opening the issue on github

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [August 20, 2019, 12:59pm UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/6 "2019-08-20T12:59:37Z")

</div>

> [@VamPikmin](#):
>
> I couldn't find the section `Using PKI to authenticate Kibana to the Elasticsearch cluster` under the blog you linked.

That section exists in the blog _you_ linked, the one you were following through.

> [@VamPikmin](#):
>
> , and follow the guide you linked.

The blog I linked is just an overview of the security features available in basic.

We do have a nice tutorial to set up security in your cluster. Read about what security means for your cluster here : [Security overview | Elasticsearch Guide [7.3] | Elastic](https://www.elastic.co/guide/en/elastic-stack-overview/7.3/elasticsearch-security.html) and you can start with the Getting Started tutorial in [Tutorial: Getting started with security | Elasticsearch Guide [7.3] | Elastic](https://www.elastic.co/guide/en/elastic-stack-overview/7.3/security-getting-started.html) that goes through authentication and [Configure TLS | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html) that discusses communications encryption

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 20, 2019, 10:31pm UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/7 "2019-08-20T22:31:25Z")

</div>

Thanks for your help once more

I've finished setting up elasticsearch part but stuck on encrypting communications in Kibana

> **[Encrypt TLS communications in Kibana | Kibana Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/kibana/current/configuring-tls.html)**

I don't see any clear examples but trying to understand what you have written here

> [@Certificates and keys for Kibana and Logstash with X-Pack](https://discuss.elastic.co/t/certificates-and-keys-for-kibana-and-logstash-with-x-pack/150390):
>
> Hello there, I'm setting up the ELK security using X-Pack, I generated the CA and Certs as suggested by the docs: bin/elasticsearch-certutil ca bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 Shipped them to all the elasticseach nodes and worked fine. On Kibana, per the document: [https://www.elastic.co/guide/en/kibana/6.3/configuring-tls.html](https://www.elastic.co/guide/en/kibana/6.3/configuring-tls.html) It says: " Generate a server certificate for Kibana. You must either set the certificate’s subjectAltName to the hostname, fully-quali…

I'm still only using localhost and nginx to proxy 5601

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 21, 2019, 2:59am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/8 "2019-08-21T02:59:47Z")

</div>

So far nothing I tried so far has worked. I followed your suggestion from the link in the previous post

EXPORT CA Certificate

`openssl pkcs12 -in /etc/elasticsearch/certs/elastic-stack-ca.p12 -clcerts -nokeys -chain -out /etc/kibana/certs/ca.pem`

EXPORT Cert and Key for Kibana

`/usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --dns elastic18`

kibana.yml

```
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "elasticsearch.password"

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/instance.crt
server.ssl.key: /etc/kibana/certs/instance.key

elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca.pem
elasticsearch.ssl.verificationMode: certificate

```

Error from Kibana

> Aug 21 12:53:31 elastic18 kibana[58413]: {"type":"error","@timestamp":"2019-08-21T02:53:31Z","tags":["connection","client","error"],"pid":58413,"level":"error","error":{"message":"140001888880448:error:1408F09C:SSL routines:ssl3\_get\_record:http request:../deps/openssl/openssl/ssl/record/ssl3\_record.c:242:\n","name":"Error","stack":"Error: 140001888880448:error:1408F09C:SSL routines:ssl3\_get\_record:http request:../deps/openssl/openssl/ssl/record/ssl3\_record.c:242:\n"},"message":"140001888880448:error:1408F09C:SSL routines:ssl3\_get\_record:http request:../deps/openssl/openssl/ssl/record/ssl3\_record.c:242:\n"}

If I comment out the # server.ssl.enabled: true  
It's working but that means ssl is not enabled doesn't it?

Thanks for all your help

EDIT:

Changed the nginx proxy\_pass directive from http to https and seems okay now

> location / {  
> proxy\_pass [https://localhost:5601](https://localhost:5601);

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 22, 2019, 12:59am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/9 "2019-08-22T00:59:34Z")

</div>

Trying to get the logstash part going but can't get rid of this error

> Manticore::UnknownException: Host name 'localhost' does not match the certificate subject provided by the peer (CN=instance)\>

I've read one of your posts and you talk about making sure to set the --dns properly which I thought I was using correctly

I've tried to set it to localhost, then I tried instance then I tried ips as well

> /usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --dns instance,localhost,elastic18 --name instance --ip 127.0.0.1,192.168.131.140 --out /etc/kibana/certs/kibana.zip

```
openssl x509 -noout -checkhost localhost -in instance.crt 
Hostname localhost does match certificate
openssl x509 -noout -checkhost instance -in instance.crt 
Hostname instance does match certificate
openssl x509 -noout -checkhost elastic18 -in instance.crt 
Hostname elastic18 does match certificate

```

Any suggestions Ioannis?  
I appreciate the help

i've been looking at this guide as well, to understand the process

> **[Configuring SSL, TLS, and HTTPS to secure Elasticsearch, Kibana, Beats, and...](https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash)**
>
> Feeling insecure about your Elastic Stack security? Run through these step-by-step instructions for setting up TLS encryption and https on Elasticsearch, Kibana, Logstash, and Beats to shore up your stack's defenses. Highly recommended for end-to-end...

This is the node name in logstash.yml

`node.name: localhost`

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [August 22, 2019, 6:50am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/10 "2019-08-22T06:50:04Z")

</div>

These are the certificates that you generate for Kibana. Logstash connects to Elasticsearch, not Kibana , and as such you need to make sure that the certificate that Elasticsearch uses for TLS has the right SANs.

The certificate that Logstash has an issue with, is the certificate that is defined by

```auto
xpack.security.http.ssl.keystore.path

```

or

```auto
xpack.security.http.ssl.certificate

```

depending on how you have configured Elasticsearch

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 22, 2019, 9:04am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/11 "2019-08-22T09:04:37Z")

</div>

Thank you so much!!!

Like you said

```
openssl pkcs12 -nokeys -info -in elastic-certificates.p12 

Certificate bag
Bag Attributes
    friendlyName: instance
    localKeyID: 54 69 6D 65 20 31 35 36 36 31 37 39 36 32 38 37 32 32 
subject=CN = instance

issuer=CN = Elastic Certificate Tool Autogenerated CA

```

I recreated the certificate

```
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --name localhost --dns localhost --ip 127.0.0.1 --out /etc/elasticsearch/certs/elastic-certificates.p12

```

I am getting this warning, but not sure if it's false

```
2019-08-22T18:57:45,941][WARN][logstash.outputs.elasticsearch] **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true

```

Maybe it applies to this

> <https://github.com/elastic/logstash/issues/10352>

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [August 22, 2019, 9:27am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/12 "2019-08-22T09:27:22Z")

</div>

> [@VamPikmin](#):
>
> I am getting this warning,

If you have set

```auto
:ssl_certificate_verification => false

```

in your Logstash's Elasticsearch output plugin configuration then you need to remove that line. If not, this is probably the bug you have referenced and you can feel free to ignore

---

<div class="post-metadata">

**Author:** ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\
**Post date:** [August 22, 2019, 11:13am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/13 "2019-08-22T11:13:52Z")

</div>

I haven't set it, and I've also tried setting it to true, didn't make a difference. Warning was still present.  
I'll ignore it and keep going with the rest of the setup.

You've helped me immensely, once again thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 19, 2019, 11:13am UTC](https://discuss.elastic.co/t/elasticsearch-security-pki/195838/14 "2019-09-19T11:13:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.