# Enable xpack on a running Elastic cluster

**URL:** <https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [September 21, 2021, 4:11am UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671 "2021-09-21T04:11:11Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ls4074](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@ls4074](https://discuss.elastic.co/u/ls4074)\
**Post date:** [September 21, 2021, 4:11am UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/1 "2021-09-21T04:11:11Z")

</div>

I deployed Elastic using helm chart (7.13) without xpack enabled. The ES and Kibana pods are running fine. I logged in to one of the master pods and ran the command "Elasticsearch-certutil" to generate the certs and then created a secret with the certs. Then I enabled xpack for the internode TLS encrypted communication. I set the following in the values.yaml file. But after ran "helm upgrade", I see the master pod is in not ready state. I have two master pods. One is in not ready state, the other is in ready state but not restarted. If I manaually restarted the other pod, then both pods are in not ready state. Is this a bug in Elastic or did I miss anything?

xpack.security.enabled: true  
xpack.security.transport.ssl.enabled: true  
xpack.security.transport.ssl.verification\_mode: certificate  
xpack.security.transport.ssl.keystore.path: /usr/share/Elasticsearch/config/certs/elastic-certificates.p12  
xpack.security.transport.ssl.truststore.path: /usr/share/Elasticsearch/config/certs/elastic-certificates.p12  
xpack.security.transport.ssl.keystore.password:   
xpack.security.transport.ssl.truststore.password:   
extraEnvs:

- name: ELASTIC\_PASSWORD  
valueFrom:  
secretKeyRef:  
name: elastic-credentials  
key: password
- name: ELASTIC\_USERNAME  
valueFrom:  
secretKeyRef:  
name: elastic-credentials  
key: username  
secretMounts:
  - name: elastic-certificates  
secretName: elastic-certificates  
path: /usr/share/Elasticsearch/config/certs

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2021, 4:15am UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/2 "2021-09-21T04:15:05Z")

</div>

What do the Elasticsearch logs show?

---

<div class="post-metadata">

**Author:** ![ls4074](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@ls4074](https://discuss.elastic.co/u/ls4074)\
**Post date:** [September 21, 2021, 2:55pm UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/3 "2021-09-21T14:55:26Z")

</div>

Below are logs in the ES master pods. BTW, I only added the xpack on the values.yaml file for the master nodes. Maybe I also need to add the xpack on the data.yaml file for the ES data nodes?

{"type": "server", "timestamp": "2021-09-21T14:50:30,188Z", "level": "WARN", "component": "o.e.x.c.s.t.n.SecurityNetty4Transport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.8.224:9300, remoteAddress=/10.0.9.24:48224, profile=default}", "cluster.uuid": "3vMs0RouRtWdD4tiT\_wAWQ", "node.id": "n8bkpWAEQJm1LIKFx8rhwQ" }

kubectl describe pod elasticsearch-master-0 -n elk  
Name: elasticsearch-master-0  
Namespace: elk  
Priority: 0  
PriorityClassName:   
Node: aks-elkmstrpool-22444732-vmss000002/10.0.8.223  
Start Time: Tue, 21 Sep 2021 09:42:24 -0500  
Labels: app=elasticsearch-master  
chart=elasticsearch  
controller-revision-hash=elasticsearch-master-7f84d46f56  
release=elasticsearch  
[statefulset.kubernetes.io/pod-name=elasticsearch-master-0](http://statefulset.kubernetes.io/pod-name=elasticsearch-master-0)  
Annotations: configchecksum=a9664c8bf0fb650cba59aa2de53d188e5f0ca9b3cb2dd1de073468cca66cebe  
Status: Running  
IP: 10.0.8.224  
Controlled By: StatefulSet/elasticsearch-master  
Init Containers:  
configure-sysctl:  
Container ID: containerd://6e89728b8dd3bde7ce0d8d4bf5f0cbfcbd72b8350420caf753ae1287a81a3a53  
Image: [docker.elastic.co/elasticsearch/elasticsearch:7.13.2](http://docker.elastic.co/elasticsearch/elasticsearch:7.13.2)  
Image ID: [docker.elastic.co/elasticsearch/elasticsearch@sha256:1cecc2c7419a4f917a88c83180335bd491d623f28ac43ca7e0e69b4eca25fbd5](http://docker.elastic.co/elasticsearch/elasticsearch@sha256:1cecc2c7419a4f917a88c83180335bd491d623f28ac43ca7e0e69b4eca25fbd5)  
Port:   
Host Port:   
Command:  
sysctl  
-w  
vm.max\_map\_count=262144  
State: Terminated  
Reason: Completed  
Exit Code: 0  
Started: Tue, 21 Sep 2021 09:42:25 -0500  
Finished: Tue, 21 Sep 2021 09:42:25 -0500  
Ready: True  
Restart Count: 0  
Environment:   
Mounts:  
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4qcbv (ro)  
Containers:  
elasticsearch:  
Container ID: containerd://0b76bc77dee4ec447b73864312d689bbe9e83ae4c50e0992810c75206e534f84  
Image: [docker.elastic.co/elasticsearch/elasticsearch:7.13.2](http://docker.elastic.co/elasticsearch/elasticsearch:7.13.2)  
Image ID: [docker.elastic.co/elasticsearch/elasticsearch@sha256:1cecc2c7419a4f917a88c83180335bd491d623f28ac43ca7e0e69b4eca25fbd5](http://docker.elastic.co/elasticsearch/elasticsearch@sha256:1cecc2c7419a4f917a88c83180335bd491d623f28ac43ca7e0e69b4eca25fbd5)  
Ports: 9200/TCP, 9300/TCP  
Host Ports: 0/TCP, 0/TCP  
State: Running  
Started: Tue, 21 Sep 2021 09:42:26 -0500  
Ready: False  
Restart Count: 0  
Limits:  
cpu: 1  
memory: 2Gi  
Requests:  
cpu: 100m  
memory: 500Mi  
Readiness: exec [sh -c #!/usr/bin/env bash -e

# If the node is starting up wait for the cluster to be ready (request params: "wait\_for\_status=green&timeout=1s" )

# Once it has started only check that the node itself is responding

START\_FILE=/tmp/.es\_start\_file

# Disable nss cache to avoid filling dentry cache when calling curl

# This is required with Elasticsearch Docker using nss \< 3.52

export NSS\_SDB\_USE\_CACHE=no

http () {  
local path="${1}"  
local args="${2}"  
set -- -XGET -s

if ["$args" != ""]; then  
set -- "$@" $args  
fi

if [-n "${ELASTIC\_USERNAME}"] && [-n "${ELASTIC\_PASSWORD}"]; then  
set -- "$@" -u "${ELASTIC\_USERNAME}:${ELASTIC\_PASSWORD}"  
fi

curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}"  
}

if [-f "${START\_FILE}"]; then  
echo 'Elasticsearch is already running, lets check the node is healthy'  
HTTP\_CODE=$(http "/" "-w %{http\_code}")  
RC=$?  
if [[${RC} -ne 0]]; then  
echo "curl --output /dev/null -k -XGET -s -w '%{http\_code}' ${BASIC\_AUTH} [http://127.0.0.1:9200/](http://127.0.0.1:9200/) failed with RC ${RC}"  
exit ${RC}  
fi

# ready if HTTP code 200, 503 is tolerable if ES version is 6.x

if [[${HTTP\_CODE} == "200"]]; then  
exit 0  
elif [[${HTTP\_CODE} == "503" && "7" == "6"]]; then  
exit 0  
else  
echo "curl --output /dev/null -k -XGET -s -w '%{http\_code}' ${BASIC\_AUTH} [http://127.0.0.1:9200/](http://127.0.0.1:9200/) failed with HTTP code ${HTTP\_CODE}"  
exit 1  
fi

else  
echo 'Waiting for elasticsearch cluster to become ready (request params: "wait\_for\_status=green&timeout=1s" )'  
if http "/\_cluster/health?wait\_for\_status=green&timeout=1s" "--fail" ; then  
touch ${START\_FILE}  
exit 0  
else  
echo 'Cluster is not yet ready (request params: "wait\_for\_status=green&timeout=1s" )'  
exit 1  
fi  
fi  
] delay=10s timeout=5s period=10s #success=3 #failure=3  
Environment:  
node.name: elasticsearch-master-0 (v1:metadata.name)  
cluster.initial\_master\_nodes: elasticsearch-master-0,elasticsearch-master-1,  
discovery.seed\_hosts: elasticsearch-master-headless  
cluster.name: elasticsearch  
network.host: 0.0.0.0  
node.data: false  
node.ingest: true  
node.master: true  
node.ml: true  
node.remote\_cluster\_client: true  
ELASTIC\_PASSWORD: \<set to the key 'password' in secret 'elastic-credentials'\> Optional: false  
ELASTIC\_USERNAME: \<set to the key 'username' in secret 'elastic-credentials'\> Optional: false  
Mounts:  
/usr/share/elasticsearch/config/certs from elastic-certificates (rw)  
/usr/share/elasticsearch/config/elasticsearch.yml from esconfig (rw)  
/usr/share/elasticsearch/data from elasticsearch-master (rw)  
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4qcbv (ro)  
Conditions:  
Type Status  
Initialized True  
Ready False  
ContainersReady False  
PodScheduled True  
Volumes:  
elasticsearch-master:  
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)  
ClaimName: elasticsearch-master-elasticsearch-master-0  
ReadOnly: false  
elastic-certificates:  
Type: Secret (a volume populated by a Secret)  
SecretName: elastic-certificates  
Optional: false  
esconfig:  
Type: ConfigMap (a volume populated by a ConfigMap)  
Name: elasticsearch-master-config  
Optional: false  
default-token-4qcbv:  
Type: Secret (a volume populated by a Secret)  
SecretName: default-token-4qcbv  
Optional: false  
QoS Class: Burstable  
Node-Selectors: nodepool=elkmaster  
Tolerations: dedicated=elkmaster:NoSchedule  
[node.kubernetes.io/memory-pressure:NoSchedule](http://node.kubernetes.io/memory-pressure:NoSchedule)  
[node.kubernetes.io/not-ready:NoExecute](http://node.kubernetes.io/not-ready:NoExecute) for 300s  
[node.kubernetes.io/unreachable:NoExecute](http://node.kubernetes.io/unreachable:NoExecute) for 300s  
Events:  
Type Reason Age From Message

* * *

Normal Scheduled Successfully assigned elk/elasticsearch-master-0 to aks-elkmstrpool-22444732-vmss000002  
Normal Pulled 11m kubelet, aks-elkmstrpool-22444732-vmss000002 Container image "[docker.elastic.co/elasticsearch/elasticsearch:7.13.2](http://docker.elastic.co/elasticsearch/elasticsearch:7.13.2)" already present on machine  
Normal Created 11m kubelet, aks-elkmstrpool-22444732-vmss000002 Created container configure-sysctl  
Normal Started 11m kubelet, aks-elkmstrpool-22444732-vmss000002 Started container configure-sysctl  
Normal Pulled 11m kubelet, aks-elkmstrpool-22444732-vmss000002 Container image "[docker.elastic.co/elasticsearch/elasticsearch:7.13.2](http://docker.elastic.co/elasticsearch/elasticsearch:7.13.2)" already present on machine  
Normal Created 11m kubelet, aks-elkmstrpool-22444732-vmss000002 Created container elasticsearch  
Normal Started 11m kubelet, aks-elkmstrpool-22444732-vmss000002 Started container elasticsearch  
Warning Unhealthy 1m (x60 over 11m) kubelet, aks-elkmstrpool-22444732-vmss000002 Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait\_for\_status=green&timeout=1s" )  
Cluster is not yet ready (request params: "wait\_for\_status=green&timeout=1s" )

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2021, 9:30pm UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/4 "2021-09-21T21:30:28Z")

</div>

> [@ls4074](#):
>
> I only added the xpack on the values.yaml file for the master nodes. Maybe I also need to add the xpack on the data.yaml file for the ES data nodes?

Yes, it needs to be done on _all_ nodes.

Please also format your code/logs/config using the `</>` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂

---

<div class="post-metadata">

**Author:** ![ls4074](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@ls4074](https://discuss.elastic.co/u/ls4074)\
**Post date:** [September 22, 2021, 12:05am UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/5 "2021-09-22T00:05:44Z")

</div>

Thank you Mark for the quick response! After I added the xpack on the ES data yaml file and deployed it, both the master pods and data pods are running and in ready state.

I have another question here. You know I first deployed the ES helm chart without xpack enabled, And I was able to login to one of the master pods and ran the Elasticsearch-certutil to generate the certs. What if I want to enable xpack at the first place, is there a way to generate the certs without logging to the master pod?

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 22, 2021, 12:07am UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/6 "2021-09-22T00:07:16Z")

</div>

I am not sure how to do that with helm sorry ☹

---

<div class="post-metadata">

**Author:** ![ls4074](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@ls4074](https://discuss.elastic.co/u/ls4074)\
**Post date:** [September 22, 2021, 11:11pm UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/7 "2021-09-22T23:11:32Z")

</div>

Thank you again for the quick response! As you see in the values.yaml file, I have the following lines:  
xpack.security.transport.ssl.keystore.password: `<passwd>`  
xpack.security.transport.ssl.truststore.password: `<passwd>`

This is not secure. So I logged in to the ES pods and ran the following commands to store the keystore and truststore passwords :

./elasticcsearch-keystore add xpack.security.transport.ssl.keystore.secure\_password  
./elasticcsearch-keystore add xpack.security.transport.ssl.truststore.secure\_password

Then reran the helm deployment. After the reran the helm deployment. the ES pods are all crashed. The log shows the following. I think the reason is rerunning helm wiped off the passwd stored in the Elasticsearch-keystore. So the question is how can I store the keystore/truststore password? Maybe create a secret and use secret mount in the values.yaml file?

Here are the error from the ES log:

```auto
ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

```

---

<div class="post-metadata">

**Author:** ![ls4074](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@ls4074](https://discuss.elastic.co/u/ls4074)\
**Post date:** [September 24, 2021, 4:00pm UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/8 "2021-09-24T16:00:42Z")

</div>

I ran the command "./elasticcsearch-keystore add xpack.security.transport.ssl.keystore.secure\_password", then  
I created a secret called "eskeystore" based on the Elasticsearch.keystore file and then upadted the values.yaml file with the following:

keystore:  
- secretName: eskeystore

after restarted the ES nodes, I see the following errors:

java.lang.IllegalArgumentException: unknown secure setting [Elasticsearch.keystore] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

the ES pod shows the following:

Mounts:  
/usr/share/Elasticsearch/config/Elasticsearch.keystore from keystore (rw)  
.....

Volumes:  
keystore:  
Type: EmptyDir (a temporary directory that shares a pod's lifetime)  
Medium:  
keystore-eskeystore:  
Type: Secret (a volume populated by a Secret)  
SecretName: eskeystore  
Optional: false  
.....

So why the volume of "keystore" is empty dir ? Did I miss anything?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 27, 2021, 12:14am UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/9 "2021-09-27T00:14:06Z")

</div>

Please format your code/logs/config using the `</>` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂

It'd also be good to show your full commands (with redactions on passwords), as well as the output of those, and the entire Elasticsearch log.

---

<div class="post-metadata">

**Author:** ![ls4074](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@ls4074](https://discuss.elastic.co/u/ls4074)\
**Post date:** [September 27, 2021, 5:28pm UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/10 "2021-09-27T17:28:59Z")

</div>

Hi Mark,

I post another topic which requires to solve first, then we can get back to this one.  
Could you please take a look at the following topic?

> [@Enable secure connection from Kibana to Elasticsearch](https://discuss.elastic.co/t/enable-secure-connection-from-kibana-to-elasticsearch/285255):
>
> I have xpack enabled on Elasticsearch (ES) on both http layer and transport layer. Now I am trying to enable the secure connection from Kibana to ES. I have added these in Kibana.yaml elasticsearchHosts: "[https://elasticsearch-master:9200](https://elasticsearch-master:9200)" Elasticsearch.ssl.certificateAuthorities: /usr/share/kibana/config/certs/elastic-stack-ca.pem secretMounts: name: elastic-ca-pem secretName: elastic-ca-pem path: /usr/share/kibana/config/certs BTW, I created the secret elastic-ca-pem based on elastic-s…

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2021, 5:29pm UTC](https://discuss.elastic.co/t/enable-xpack-on-a-running-elastic-cluster/284671/11 "2021-10-25T17:29:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.