# Elasticsearch-create-enrollment-token is not possible without a keystore ( aka with PEM certificates) **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [June 15, 2023, 7:06pm UTC](https://discuss.elastic.co/t/elasticsearch-create-enrollment-token-is-not-possible-without-a-keystore-aka-with-pem-certificates/336136 "2023-06-15T19:06:51Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![DavidDPD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daviddpd/32/122317_2.png) [@DavidDPD](https://discuss.elastic.co/u/DavidDPD)\ **Post date:** [June 15, 2023, 7:06pm UTC](https://discuss.elastic.co/t/elasticsearch-create-enrollment-token-is-not-possible-without-a-keystore-aka-with-pem-certificates/336136/1 "2023-06-15T19:06:51Z")
This has been posted a few times, but threads have been auto-closed without an explicit explanation. I'm posting this as this really either needs to be changed/fixed in Elasticsearch, or DOCUMENTED. It does not seem to be documented, nor posted in the forum that it is not currently possible to use token enrollment with PEM certificates. Not having a keystore will make it impossible to do to token enrollment. At least this is my unconfirmed assertion. ```auto ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore ``` It is in the documentation, that whenever one uses `xpack.security.*.ssl.[key|certificate|certificate_authorities]` is mutually exclusive with `xpack.security.*.ssl.[keystore|truststore]`. And Elasticsearch will work well with, per say, Let's Encrypt (or other ACME distributed certificates) for both TRANSPORT (internode, 9300?) and HTTP(s) (direct API, 9200? ) - however, once configuring these options this effectively breaks `xpack.security.enrollment.enabled`. So the process for initial setup of a cluster is , run all nodes in a autoconfig - with auto-generated certificates, ```auto xpack.security.enrollment.enabled: true xpack.security.autoconfiguration.enabled: true xpack.security.enabled: true path.data: /elastic/data path.logs: /var/log/elasticsearch ``` Then , use `elasticsearch-create-enrollment-token` then, enroll the other nodes via `elasticsearch --enrollment-token`. After the cluster is up, you can then switch out the certificates to self-sign PEMs. Since most things in my network use flat PEM files, the certificate management is designed for this. A one-off system to auto import these into the keystore, when they renew every ~60 days, seems to be a solution ... however super annoying. It is likely possible to use the 3rd party CA signed certificates, however, these must be in keystores, not PEM file. It doesn't look like the enrollment-token is being saved in the keystore - as the pk12 files are not modified at `create-enrollment-token` time, so this must be an artificial check. This creates an additional problem, as then when certificates are moved to PEM and keystore is disabled, then new nodes cannot be enrolled. (yes?) This really makes using PEM certificates dangerous and pointless, especially in a production environment. Can someone please actually confirm this assertion ? If not, how does one do token enrollment with PEM certificates ? Can Elastic please update their documentation? Can this be fixed ? This is seemingly arbitrary enforcement/limitation. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 15, 2023, 9:14pm UTC](https://discuss.elastic.co/t/elasticsearch-create-enrollment-token-is-not-possible-without-a-keystore-aka-with-pem-certificates/336136/2 "2023-06-15T21:14:23Z")
Hi @DavidDPD Welcome to the community, and apologies for the frustrating experience. Yes a bit frustrating and probably could be documented better. I think perhaps there is a fundamental "disconnect" At least this is how I understand it.... When you want to use your own certs etc, you are no longer using security auto configuration and thus when you use your own certs you should no longer be trying to "enroll" another node in the cluster using an enrollment token... the `elasticsearch-create-enrollment-token` tool is only used with security auto configuration. Note from the enrollment token command line docs [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/create-enrollment-token.html) > `elasticsearch-create-enrollment-token` can only be used with Elasticsearch clusters that have been [auto-configured for security](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html). Its OK if you initially did security auto-configuration you can update the cert settings and move forward. Once you have decided to use your own certs / .pems etc you should simply set up your certs and set the correct discovery and cluster formation settings. No enrollment token is necessary. > **[Important Elasticsearch configuration | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#discovery-settings)** I.e. set the discovery and initial master nodes and the same cluster name and the new nodes should join assuming the `transport.host` is set correctly and there is network connectivity. Perhaps someone else has a perspective. There is a rather long history of how Elasticsearch got to security auto-configuration in 8.x, your use case is the way we all set it up prior to 8.x ---
**Author:** ![DavidDPD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daviddpd/32/122317_2.png) [@DavidDPD](https://discuss.elastic.co/u/DavidDPD)\ **Post date:** [June 16, 2023, 2:32am UTC](https://discuss.elastic.co/t/elasticsearch-create-enrollment-token-is-not-possible-without-a-keystore-aka-with-pem-certificates/336136/3 "2023-06-16T02:32:29Z")
Ah, Thanks ... I've been sys-admin'ing Elasticsearch for 4+ years, so I go back to 7.x. I'm setting up a brand new cluster for the first time in a year or two, 8.8.1 in a new QA environment. Previously bootstrapping the cluster didn't now work with this config, so I had gone back to the "Getting Starting ..." docs - the RPM installation ... is till a bit awakard, for setting up PEM Certificate clusters. - [Install Elasticsearch with RPM | Elasticsearch Guide [8.8] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/rpm.html) However, for the sake of the next person that google's this is how to boot-strap/create a new cluster - QAW1 (QA-West 1) - with a four node cluster, using Let's Encrypt wild-card certificate on both transport and http. This is the `elasticsearch.yml` from `es1.qaw1` , and started them all at the same time with ansible ... `ansible -f 9 -u dpd [...] -a "systemctl start elasticsearch "`: ```auto cluster.name: qaw1 node.name: es1.qaw1 node.attr.site: qaw1 node.attr.host: es1 path.data: /elastic/data path.logs: /var/log/elasticsearch path: repo: - /z/elastic - /z/elastic/snapshots network.host: 10.11.162.11 transport.host: 10.11.162.11 http.host: 10.11.162.11 discovery.seed_hosts: - es1.qaw1 - es2.qaw1 - es3.qaw1 - es4.qaw1 cluster.initial_master_nodes: - es1.qaw1 - es2.qaw1 - es3.qaw1 - es4.qaw1 xpack.security.enrollment.enabled: true xpack.security.autoconfiguration.enabled: false xpack.security.enabled: true # Enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl: enabled: true verification_mode: certificate key: /etc/elasticsearch/ssl/certs/corp/key.pem certificate: /etc/elasticsearch/ssl/certs/corp/chain.pem certificate_authorities: ["/etc/elasticsearch/ssl/certs/corp/le-ca-RSA-chain.pem", "/etc/elasticsearch/ssl/certs/corp/le-ca.pem", "/etc/elasticsearch/ssl/certs/corp/ca-bundle-chain.pem"] # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents xpack.security.http.ssl: enabled: true verification_mode: certificate key: /etc/elasticsearch/ssl/certs/corp/key.pem certificate: /etc/elasticsearch/ssl/certs/corp/chain.pem certificate_authorities: ["/etc/elasticsearch/ssl/certs/corp/le-ca.pem", "/etc/elasticsearch/ssl/certs/corp/le-ca-RSA-chain.pem", "/etc/elasticsearch/ssl/certs/corp/ca-bundle-chain.pem"] ``` For those using Let's Encrypt specifically, but certificate management scripts - after the recent LE Root expiring - my CA bundles are built like this : ```auto #Intermediate Certificates R3="lets-encrypt-r3.pem" E1="lets-encrypt-e1.pem" # Root X2="isrg-root-x2.pem" X1="isrgrootx1.pem" X1XS="isrg-root-x1-cross-signed.pem" #backup R4="lets-encrypt-r4.pem" E2="" # le-ca.pem - full RSA chain. LE_CA_FILES= for c in $R3 $R4 $X2 $X1; do if [-n "${LE_CA_FILES}"]; then LE_CA_FILES="${LE_CA_FILES} " fi LE_CA_FILES="${LE_CA_FILES}${ACMEPATH_NFS}/le-root-certs/$c" done cat ${CABUNDLE} ${LE_CA_FILES} > ${CERTS}/${SRV}/ca-bundle-chain.pem.tmp cat ${LE_CA_FILES} > ${CERTS}/${SRV}/le-ca-RSA-chain.pem.tmp cat ${ACMEPATH_NFS}/le-root-certs/$R3 ${ACMEPATH_NFS}/le-root-certs/$X1 > ${CERTS}/${SRV}/le-ca.pem.tmp for capem in ca-bundle-chain.pem le-ca-RSA-chain.pem le-ca.pem; do diff ${CERTS}/${SRV}/$capem ${CERTS}/${SRV}/${capem}.tmp > /dev/null if [$? -eq 1 -o ! -f "${CERTS}/${SRV}/${capem}"]; then mv -fv ${CERTS}/${SRV}/${capem}.tmp ${CERTS}/${SRV}/${capem} fi done ``` And ... ( note, with signed LE certificates, curl built in CAs will validate the certificates, but a FQDN, matching the CN= in the certificate ... needs to be used. ) ```auto curl -u elastic: ***** -X GET "https://es1.qaw1.${FQDN}:9200/_cluster/health?pretty" { "cluster_name" : "qaw1", "status" : "green", "timed_out" : false, "number_of_nodes" : 4, "number_of_data_nodes" : 4, "active_primary_shards" : 1, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` ---
**Author:** ![merson](https://avatars.discourse-cdn.com/v4/letter/m/4af34b/32.png) [@merson](https://discuss.elastic.co/u/merson)\ **Post date:** [June 27, 2023, 7:33am UTC](https://discuss.elastic.co/t/elasticsearch-create-enrollment-token-is-not-possible-without-a-keystore-aka-with-pem-certificates/336136/4 "2023-06-27T07:33:58Z")
Thank you for your clear explanation @stephenb @DavidDPD ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2023, 7:34am UTC](https://discuss.elastic.co/t/elasticsearch-create-enrollment-token-is-not-possible-without-a-keystore-aka-with-pem-certificates/336136/5 "2023-07-25T07:34:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.