# ELK Stack + TLS + Headache

**URL:** <https://discuss.elastic.co/t/elk-stack-tls-headache/279429>\
**Category:** Kibana\
**Tags:** elastic-stack-security\
**Created:** [July 22, 2021, 11:35pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429 "2021-07-22T23:35:33Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 22, 2021, 11:35pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/1 "2021-07-22T23:35:33Z")

</div>

I am ashamed to admin that I spent a week or so trying to get SSL enabled in my stack and this is my last ditch effort. I have scoured the docs, message boards, git but I am getting nowhere.

ELK Stack version `7.13.2` installed via Helm

TLS for transport but SSL is not working for HTTP transactions.

Certs have been generated via `elasticsearch-certutil` and mounted as Kubernetes secrets.

Any breadcrumbs will be appreciated.

`elasticsearch.yaml`

```auto
esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true

    cluster.name: elasticsearch
    node.name: elasticsearch-master-0
    discovery.seed_hosts: ["127.0.0.1"]
    network.host: 0.0.0.0
    cluster.initial_master_nodes: ["elasticsearch-master-0"]

    xpack.security.transport.ssl.verification_mode: certificate 
    xpack.security.transport.ssl.client_authentication: none
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.keystore.password: changeme
    xpack.security.transport.ssl.truststore.password: changeme
    
    xpack.security.http.ssl.enabled: true
    xpack.security.authc.api_key.enabled: true
    xpack.security.http.ssl.client_authentication: none
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.password: changeme
    xpack.security.http.ssl.truststore.password: changeme
    

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    value: "changeme"
  - name: "ELASTIC_USERNAME"
    value: "changeme"

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates #elastic-certificates.p12
    path: /usr/share/elasticsearch/config/certs
    defaultMode: 0755
  - name: elastic-ca
    secretName: elastic-ca #elastic-stack-ca.p12
    path: /usr/share/elasticsearch/config/ca
    defaultMode: 0755
  - name: http-certificates
    secretName: http-certificates #elastic-certificates.p12
    path: /usr/share/elasticsearch/config/certs/http
    defaultMode: 0755
  - name: elastic-ca-pem
    secretName: elastic-ca-pem #elasticsearch-ca.pem
    path: /usr/share/kibana/config/ca/pem

```

`kibana.yaml`

```auto
kibanaConfig:
  kibana.yml: |
    elasticsearch.username: "kibana_system"
    elasticsearch.password: "changeme"
    elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/ca/elasticsearch-ca.pem"]
    elasticsearch.hosts: ["https://elasticsearch-master:9200"]
    
    xpack.encryptedSavedObjects.encryptionKey: 'd2Nuvvt6s0ZmxuBxzzTVMs/i73hjTtRZw+q8YnQxytsK'
    server.ssl.keystore.path: "/usr/share/kibana/config/certs/kibana/kibana-server.p12"
    server.ssl.keystore.password: "changeme"
    server.ssl.enabled: true
    elasticsearch.ssl.verificationMode: none

    server.name: elasticsearch-master-0

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    value: "changeme"
  - name: "ELASTIC_USERNAME"
    value: "changeme"
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates #elastic-certificates.p12
    path: /usr/share/kibana/config/certs
  - name: elastic-ca-pem
    secretName: elastic-ca-pem #elasticsearch-ca.pem
    path: /usr/share/kibana/config/ca
  - name: kibana-certificates
    secretName: kibana-certificates #kibana-server.p12
    path: /usr/share/kibana/config/certs/kibana

```

Here is the error that I am getting in Kibana. If I remove the SSL bits Kibana and ES communicate just fine.

```auto
{"type":"log","@timestamp":"2021-07-22T23:08:35+00:00","tags":["info","plugins-service"],"pid":951,"message":"Plugin \"timelines\" is disabled."}
{"type":"log","@timestamp":"2021-07-22T23:08:35+00:00","tags":["warning","config","deprecation"],"pid":951,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-07-22T23:08:35+00:00","tags":["warning","config","deprecation"],"pid":951,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-22T23:08:38+00:00","tags":["info","plugins-system"],"pid":951,"message":"Setting up [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeTagcloud,visTypeTable,visTypeVislib,visTypeVega,visTypeMetric,visTypeTimelion,features,licenseManagement,watcher,visTypeMarkdown,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,snapshotRestore,upgradeAssistant,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-22T23:08:38+00:00","tags":["info","plugins","taskManager"],"pid":951,"message":"TaskManager is identified by the Kibana UUID: efb7963f-2f7c-48f6-bed1-a7a87b6be4ac"}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["warning","plugins","security","config"],"pid":951,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["warning","plugins","reporting","config"],"pid":951,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["warning","plugins","reporting","config"],"pid":951,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.4.2105\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":951,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-22T23:08:40+00:00","tags":["info","savedobjects-service"],"pid":951,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-22T23:08:40+00:00","tags":["error","savedobjects-service"],"pid":951,"message":"Unable to retrieve version information from Elasticsearch nodes."}

```

---

<div class="post-metadata">

**Author:** ![tsullivan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsullivan/32/31077_2.png) [@tsullivan](https://discuss.elastic.co/u/tsullivan)\
**Post date:** [July 23, 2021, 11:26pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/2 "2021-07-23T23:26:03Z")

</div>

> Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations

Try changing the overall log level to `debug` to see the error messages coming from failed connections to Elasticsearch:

> **[Logging settings in Kibana | Kibana Guide \[7.13\] | Elastic](https://www.elastic.co/guide/en/kibana/7.13/logging-settings.html#change-overall-log-level)**

> [@pooch](#):
>
> ```auto
> xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
> xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
> 
> ```

Try getting into an interactive session in the running container and verify that these paths and their file permissions are valid.

Hope that helps!  
-Tim

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 26, 2021, 7:13pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/3 "2021-07-26T19:13:31Z")

</div>

Thank you for the reply!

The permissions seem okay to me. I will enable `debug` for the logging and see if that gives me a clue.

```auto
bash-4.4$ ls -la /usr/share/kibana/config/
total 20
drwxrwxr-x 1 kibana root 4096 Jul 23 00:41 .
drwxrwsr-x 1 kibana root 4096 Jun 10 20:59 ..
drwxrwsrwt 3 root kibana 100 Jul 23 00:40 ca
drwxrwsrwt 4 root kibana 120 Jul 23 00:40 certs
-rw-r--r-- 1 root kibana 546 Jul 23 00:40 kibana.yml
-rw-rw-r-- 1 kibana root 216 Jun 10 20:24 node.options

bash-4.4$ ls -la /usr/share/kibana/config/certs
total 4
drwxrwsrwt 4 root kibana 120 Jul 23 00:40 .
drwxrwxr-x 1 kibana root 4096 Jul 23 00:41 ..
drwxr-sr-x 2 root kibana 60 Jul 23 00:40 ..2021_07_23_00_40_51.279187955
lrwxrwxrwx 1 root root 31 Jul 23 00:40 ..data -> ..2021_07_23_00_40_51.279187955
lrwxrwxrwx 1 root root 31 Jul 23 00:40 elastic-certificates.p12 -> ..data/elastic-certificates.p12
drwxrwsrwt 3 root kibana 100 Jul 23 00:40 kibana

bash-4.4$ ls -la /usr/share/kibana/config/ca
total 4
drwxrwsrwt 3 root kibana 100 Jul 23 00:40 .
drwxrwxr-x 1 kibana root 4096 Jul 23 00:41 ..
drwxr-sr-x 2 root kibana 60 Jul 23 00:40 ..2021_07_23_00_40_51.091807926
lrwxrwxrwx 1 root root 31 Jul 23 00:40 ..data -> ..2021_07_23_00_40_51.091807926
lrwxrwxrwx 1 root root 27 Jul 23 00:40 elasticsearch-ca.pem -> ..data/elasticsearch-ca.pem

bash-4.4$ ls -la /usr/share/kibana/config/certs/kibana
total 0
drwxrwsrwt 3 root kibana 100 Jul 23 00:40 .
drwxrwsrwt 4 root kibana 120 Jul 23 00:40 ..
drwxr-sr-x 2 root kibana 60 Jul 23 00:40 ..2021_07_23_00_40_52.708615512
lrwxrwxrwx 1 root root 31 Jul 23 00:40 ..data -> ..2021_07_23_00_40_52.708615512
lrwxrwxrwx 1 root root 24 Jul 23 00:40 kibana-server.p12 -> ..data/kibana-server.p12
bash-4.4$

```

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [July 26, 2021, 7:52pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/4 "2021-07-26T19:52:50Z")

</div>

Hi there,

How did you get `elasticsearch-ca.pem` ? Did you export it from `elastic-stack-ca.p12` or `elastic-certificates.p12` ? If so, how ?

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 26, 2021, 10:43pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/5 "2021-07-26T22:43:36Z")

</div>

Hi!

I followed the docs and used the command below.

`openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elasticsearch-ca.pem`

> [@ELK Stack + TLS + Headache](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/3):
>
> Thank you for the reply! The permissions seem okay to me. I will enable debug for the logging and see if that gives me a clue. bash-4.4$ ls -la /usr/share/kibana/config/ total 20 drwxrwxr-x 1 kibana root 4096 Jul 23 00:41 . drwxrwsr-x 1 kibana root 4096 Jun 10 20:59 .. drwxrwsrwt 3 root kibana 100 Jul 23 00:40 ca drwxrwsrwt 4 root kibana 120 Jul 23 00:40 certs -rw-r--r-- 1 root kibana 546 Jul 23 00:40 kibana.yml -rw-rw-r-- 1 kibana root 216 Jun 10 20:24 node.options bash-4.4$ …

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 26, 2021, 10:44pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/6 "2021-07-26T22:44:41Z")

</div>

> **[Encrypt communications in Kibana | Kibana Guide \[7.13\] | Elastic](https://www.elastic.co/guide/en/kibana/current/configuring-tls.html)**

---

<div class="post-metadata">

**Author:** ![AdamWritesDocs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamwritesdocs/32/88193_2.png) [@AdamWritesDocs](https://discuss.elastic.co/u/AdamWritesDocs)\
**Post date:** [July 27, 2021, 11:53am UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/7 "2021-07-27T11:53:33Z")

</div>

@pooch, I see that you used the `openssl` command to generate that certificate, which is indicated in the Kibana documentation.

Our current SSL documentation indicates to use the `elasticsearch-certutil` command to generate security certificates. When encrypting HTTP client communications for Elasticsearch, use the `elasticsearch-certutil` command in `http` mode to create the `http.12` keystore and the `elasticsearch.pem` file. Can you follow the instructions at this link and replace your certificates with the ones generated by the `elasticsearch-certutil` command?

[https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#encrypt-http-communication](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#encrypt-http-communication)

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 27, 2021, 6:16pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/8 "2021-07-27T18:16:44Z")

</div>

Thanks for your time!

I am following the docs from the link provided.

Question...I want an elastic Elasticsearch cluster where I can add nodes (pods) as needed. Is there a way to define a wildcard cert and not get tied down to node names?

> Enter all hostnames used to connect to your first node. These hostnames will be added as DNS names in the Subject Alternative Name (SAN) field in your certificate.

> List every hostname and variant used to connect to your cluster over HTTPS.

> Enter the IP addresses that clients can use to connect to your node.  
> Repeat these steps for each additional node in your cluster.

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 27, 2021, 6:28pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/9 "2021-07-27T18:28:30Z")

</div>

Here is my cert generation process...

```auto
[elasticsearch@elasticsearch-master-0 ~]$ ./bin/elasticsearch-certutil http

## Elasticsearch HTTP Certificate Utility

The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.

This tool will ask you a number of questions in order to generate the right
set of files for your needs.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA).

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.

Generate a CSR? [y/N]n

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]y

## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
CA Path: /usr/share/elasticsearch/config/ca/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
Password for elastic-stack-ca.p12:

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 5y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]N

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

*.com
*.dev.<redacted>.<redacted>.com
*.dev.<redacted>.com
*.<redacted>.com
elasticsearch*
kibana*

You entered the following hostnames.

 - *.com
 - *.dev.<redacted>.<redacted>.com
 - *.dev.<redacted>.com
 - *.<redacted>.com
 - elasticsearch*
 - kibana*

Is this correct [Y/n]Y

## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

0.0.0.0

You entered the following IP addresses.

 - 0.0.0.0

Is this correct [Y/n]Y

## Other certificate options

The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: com
Subject DN: CN=com
Key Size: 2048

Do you wish to change any of these options? [y/N]N

## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file: [<ENTER> for none]
Repeat password to confirm:

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip

```

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 28, 2021, 12:06am UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/10 "2021-07-28T00:06:22Z")

</div>

I am still getting the error  
`{"type":"log","@timestamp":"2021-07-27T23:48:03+00:00","tags":["info","savedobjects-service"],"pid":951,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}`

Here is the status of the pods...

```auto
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 8m16s
kibana-kibana-58757c8b7-vwhwj 0/1 Running 0 87s

```

I am not concerned about the section regarding encryption between browsers and es or kibana as these endpoints will not be publicly exposed. If we do expose them we will use Let Encrypt to handle the ingress certs.

I did configure according to these sections.

> **[Set up basic security for the Elastic Stack plus secured HTTPS traffic |...](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#encrypt-http-communication)**

> **[Set up basic security for the Elastic Stack plus secured HTTPS traffic |...](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#encrypt-kibana-http)**

Here is the new config files...

```auto
esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: false
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/ca/p12/http.p12
    xpack.security.http.ssl.keystore.password: elastic
    logger.org.elasticsearch.discovery: DEBUG

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    value: "elastic"
  - name: "ELASTIC_USERNAME"
    value: "elastic"

secretMounts:
  - name: elastic-http-p12
    secretName: elastic-http-p12 #elastic-http-p12
    path: /usr/share/elasticsearch/config/ca/p12

```

```auto
kibanaConfig:
  kibana.yml: |
    elasticsearch.ssl.certificateAuthorities: /usr/share/kibana/config/ca/elasticsearch-ca.pem
    elasticsearch.hosts: "https://elasticsearch-master:9200"

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    value: "elastic"
  - name: "ELASTIC_USERNAME"
    value: "elastic"
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"

secretMounts:
  - name: elastic-ca-pem
    secretName: elastic-ca-pem #elasticsearch-ca.pem
    path: /usr/share/kibana/config/ca

```

---

<div class="post-metadata">

**Author:** ![AdamWritesDocs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamwritesdocs/32/88193_2.png) [@AdamWritesDocs](https://discuss.elastic.co/u/AdamWritesDocs)\
**Post date:** [July 28, 2021, 3:47pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/11 "2021-07-28T15:47:27Z")

</div>

@pooch, in your `elasticsearch.yml` file, I see `xpack.security.enabled: false`. That value should be `true` to enable security.

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [July 28, 2021, 9:52pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/12 "2021-07-28T21:52:53Z")

</div>

Here are the steps that I went through today.

1. In `elasticsearch.yml` I set the following.  
`xpack.security.enabled: true`

2. Deployed the ES Helm chart and this error was thrown.  
`bootstrap check failure [1] of [1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]`

3. Enabled `xpack.security.transport.ssl.enabled` and the ES pod successfully scheduled but Kibana still is not being scheduled and I am getting the errors below.

```auto
{"type": "server", "timestamp": "2021-07-28T21:29:08,813Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.16.0.45:9200, remoteAddress=/172.16.0.92:47014}", "cluster.uuid": "QL1igfnYQwieDUcOXcehVg", "node.id": "3kBaDw1cSaSgsvVE4tmJbA" }

```

```auto
(⎈ |dev-kube10-admin:elk)base ❯ k get po
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 12m
kibana-kibana-58757c8b7-vwhwj 0/1 Running 0 21h

```

1. Rolled the `kibana` deployment and I am still seeing the issue.

```auto
{"type":"log","@timestamp":"2021-07-28T21:48:56+00:00","tags":["info","savedobjects-service"],"pid":951,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-28T21:48:56+00:00","tags":["error","savedobjects-service"],"pid":951,"message":"Unable to retrieve version information from Elasticsearch nodes."}

```

---

<div class="post-metadata">

**Author:** ![AdamWritesDocs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamwritesdocs/32/88193_2.png) [@AdamWritesDocs](https://discuss.elastic.co/u/AdamWritesDocs)\
**Post date:** [July 29, 2021, 4:11pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/13 "2021-07-29T16:11:58Z")

</div>

@pooch, I reached out to our Kubernetes folks and they provided the following resources that might be helpful in troubleshooting. I'm not familiar with our k8s deployments, but I hope that these examples provide some insight into resolving your connection issues.

> **[helm-charts/elasticsearch at 7.x · elastic/helm-charts - How to deploy clusters with security (authentication and TLS) enabled?](https://github.com/elastic/helm-charts/tree/7.x/elasticsearch#how-to-deploy-clusters-with-security-authentication-and-tls-enabled)**
>
> 7.x/elasticsearch

> **[helm-charts/kibana at 7.x · elastic/helm-charts - How to use Kibana with security (authentication and TLS) enabled?](https://github.com/elastic/helm-charts/tree/7.x/kibana#how-to-use-kibana-with-security-authentication-and-tls-enabled)**
>
> 7.x/kibana

---

<div class="post-metadata">

**Author:** ![AdamWritesDocs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamwritesdocs/32/88193_2.png) [@AdamWritesDocs](https://discuss.elastic.co/u/AdamWritesDocs)\
**Post date:** [August 17, 2021, 12:17pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/14 "2021-08-17T12:17:01Z")

</div>

@pooch, I'm just circling back to see if the links that I provided helped to resolve your issue. Please let us know if you were able to move forward!

---

<div class="post-metadata">

**Author:** ![pooch](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@pooch](https://discuss.elastic.co/u/pooch)\
**Post date:** [August 20, 2021, 6:39pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/15 "2021-08-20T18:39:42Z")

</div>

Thanks Adam! The examples did give me some clues. What I ended up doing was using `cert manager` to create a CA Cluster Issuer and created certs for all of my ELK services and referenced my CA in the key `certificateAuthorities`. All is working now!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 17, 2021, 6:40pm UTC](https://discuss.elastic.co/t/elk-stack-tls-headache/279429/16 "2021-09-17T18:40:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.