Elasticsearch-create-enrollment-token is not possible without a keystore ( aka with PEM certificates)

Elastic Stack Elasticsearch
1

This has been posted a few times, but threads have been auto-closed without an explicit explanation.

I'm posting this as this really either needs to be changed/fixed in Elasticsearch, or DOCUMENTED. It does not seem to be documented, nor posted in the forum that it is not currently possible to use token enrollment with PEM certificates. Not having a keystore will make it impossible to do to token enrollment. At least this is my unconfirmed assertion.

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore

It is in the documentation, that whenever one uses xpack.security.*.ssl.[key|certificate|certificate_authorities] is mutually exclusive with xpack.security.*.ssl.[keystore|truststore]. And Elasticsearch will work well with, per say, Let's Encrypt (or other ACME distributed certificates) for both TRANSPORT (internode, 9300?) and HTTP(s) (direct API, 9200? ) - however, once configuring these options this effectively breaks xpack.security.enrollment.enabled.

So the process for initial setup of a cluster is , run all nodes in a autoconfig - with auto-generated certificates,

xpack.security.enrollment.enabled: true
xpack.security.autoconfiguration.enabled: true
xpack.security.enabled: true
path.data: /elastic/data
path.logs: /var/log/elasticsearch

Then , use elasticsearch-create-enrollment-token then, enroll the other nodes via elasticsearch --enrollment-token. After the cluster is up, you can then switch out the certificates to self-sign PEMs.

Since most things in my network use flat PEM files, the certificate management is designed for this. A one-off system to auto import these into the keystore, when they renew every ~60 days, seems to be a solution ... however super annoying. It is likely possible to use the 3rd party CA signed certificates, however, these must be in keystores, not PEM file.

It doesn't look like the enrollment-token is being saved in the keystore - as the pk12 files are not modified at create-enrollment-token time, so this must be an artificial check.

This creates an additional problem, as then when certificates are moved to PEM and keystore is disabled, then new nodes cannot be enrolled. (yes?)

This really makes using PEM certificates dangerous and pointless, especially in a production environment.

Can someone please actually confirm this assertion ?

If not, how does one do token enrollment with PEM certificates ?

Can Elastic please update their documentation?

Can this be fixed ? This is seemingly arbitrary enforcement/limitation.

2

Hi @DavidDPD Welcome to the community, and apologies for the frustrating experience.

Yes a bit frustrating and probably could be documented better.

I think perhaps there is a fundamental "disconnect"

At least this is how I understand it....

When you want to use your own certs etc, you are no longer using security auto configuration and thus when you use your own certs you should no longer be trying to "enroll" another node in the cluster using an enrollment token... the elasticsearch-create-enrollment-token tool is only used with security auto configuration.

Note from the enrollment token command line docs here

elasticsearch-create-enrollment-token can only be used with Elasticsearch clusters that have been auto-configured for security.

Its OK if you initially did security auto-configuration you can update the cert settings and move forward.

Once you have decided to use your own certs / .pems etc you should simply set up your certs and set the correct discovery and cluster formation settings. No enrollment token is necessary.

I.e. set the discovery and initial master nodes and the same cluster name and the new nodes should join assuming the transport.host is set correctly and there is network connectivity.

Perhaps someone else has a perspective.

There is a rather long history of how Elasticsearch got to security auto-configuration in 8.x, your use case is the way we all set it up prior to 8.x

3

Ah, Thanks ... I've been sys-admin'ing Elasticsearch for 4+ years, so I go back to 7.x. I'm setting up a brand new cluster for the first time in a year or two, 8.8.1 in a new QA environment. Previously bootstrapping the cluster didn't now work with this config, so I had gone back to the "Getting Starting ..." docs - the RPM installation ... is till a bit awakard, for setting up PEM Certificate clusters.

However, for the sake of the next person that google's this is how to boot-strap/create a new cluster - QAW1 (QA-West 1) - with a four node cluster, using Let's Encrypt wild-card certificate on both transport and http.

This is the elasticsearch.yml from es1.qaw1 , and started them all at the same time with ansible ... ansible -f 9 -u dpd [ ...] -a "systemctl start elasticsearch ":

cluster.name: qaw1
node.name: es1.qaw1
node.attr.site: qaw1
node.attr.host: es1


path.data: /elastic/data
path.logs: /var/log/elasticsearch
path:
  repo:
    - /z/elastic
    - /z/elastic/snapshots
network.host: 10.11.162.11
transport.host: 10.11.162.11
http.host: 10.11.162.11

discovery.seed_hosts:
    - es1.qaw1
    - es2.qaw1
    - es3.qaw1
    - es4.qaw1
cluster.initial_master_nodes:
    - es1.qaw1
    - es2.qaw1
    - es3.qaw1
    - es4.qaw1

xpack.security.enrollment.enabled: true
xpack.security.autoconfiguration.enabled: false
xpack.security.enabled: true


# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
    enabled: true
    verification_mode: certificate
    key: /etc/elasticsearch/ssl/certs/corp/key.pem
    certificate: /etc/elasticsearch/ssl/certs/corp/chain.pem
    certificate_authorities: [ "/etc/elasticsearch/ssl/certs/corp/le-ca-RSA-chain.pem", "/etc/elasticsearch/ssl/certs/corp/le-ca.pem", "/etc/elasticsearch/ssl/certs/corp/ca-bundle-chain.pem" ]

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
    enabled: true
    verification_mode: certificate
    key:  /etc/elasticsearch/ssl/certs/corp/key.pem
    certificate: /etc/elasticsearch/ssl/certs/corp/chain.pem
    certificate_authorities: [ "/etc/elasticsearch/ssl/certs/corp/le-ca.pem", "/etc/elasticsearch/ssl/certs/corp/le-ca-RSA-chain.pem", "/etc/elasticsearch/ssl/certs/corp/ca-bundle-chain.pem" ]

For those using Let's Encrypt specifically, but certificate management scripts - after the recent LE Root expiring - my CA bundles are built like this :

	#Intermediate Certificates

	R3="lets-encrypt-r3.pem"
	E1="lets-encrypt-e1.pem"

	# Root
	X2="isrg-root-x2.pem"
	X1="isrgrootx1.pem"
	X1XS="isrg-root-x1-cross-signed.pem"

	#backup
	R4="lets-encrypt-r4.pem"
	E2=""
	

    # le-ca.pem - full RSA chain.
    LE_CA_FILES=
    for c in $R3 $R4 $X2 $X1; do
        if [ -n "${LE_CA_FILES}" ]; then
            LE_CA_FILES="${LE_CA_FILES} "
        fi
        LE_CA_FILES="${LE_CA_FILES}${ACMEPATH_NFS}/le-root-certs/$c"
    done
    cat ${CABUNDLE} ${LE_CA_FILES} > ${CERTS}/${SRV}/ca-bundle-chain.pem.tmp
    cat ${LE_CA_FILES} > ${CERTS}/${SRV}/le-ca-RSA-chain.pem.tmp
    cat ${ACMEPATH_NFS}/le-root-certs/$R3 ${ACMEPATH_NFS}/le-root-certs/$X1 > ${CERTS}/${SRV}/le-ca.pem.tmp

    for capem in ca-bundle-chain.pem le-ca-RSA-chain.pem le-ca.pem; do
        diff ${CERTS}/${SRV}/$capem ${CERTS}/${SRV}/${capem}.tmp  > /dev/null
        if [ $? -eq 1 -o ! -f "${CERTS}/${SRV}/${capem}" ]; then
            mv -fv ${CERTS}/${SRV}/${capem}.tmp ${CERTS}/${SRV}/${capem}
        fi
    done

And ... ( note, with signed LE certificates, curl built in CAs will validate the certificates, but a FQDN, matching the CN= in the certificate ... needs to be used. )

curl -u elastic:*****  -X GET "https://es1.qaw1.${FQDN}:9200/_cluster/health?pretty"
{
  "cluster_name" : "qaw1",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
2 Likes
4

Thank you for your clear explanation @stephenb
@DavidDPD

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.