How to automatically create a new Elasticsearch cluster (via Terraform and user-data scripts)?

[burnersk]

I am about to create a small Terraform script for an Elastic Stack "training" (not an actual training, just infrastructure description for a playground cluster), which setups Elasticsearch using a Cloud-init user-data script. This is not a configuration for an internet/intranet-connected environment (allowing any/random nodes to join the cluster without authentication/authorization is dangerous)!

The Terraform script deploys 3 Elasticsearch nodes (1 master node incl. other non-data roles, 2 data nodes) without public IP addresses and with an internal private network subnet (e.g., 10.0.0.0/29). The IP address will be defined within the Terraform script (e.g., es1 is 10.0.0.2, es2 is 10.0.0.3, ...). There is no security risk regarding rogue nodes joining and stealing data.

The setup of es1 (master) looks like the following:

http.host: 0.0.0.0
http.port: 9200
transport.host: _site_ # will bind to 10.0.0.2 (es1)
transport.port: 9300
node.name: "es1"
node.roles: [ master, ingest, ml, transform ]
cluster.name: elasticstack
discovery.seed_hosts: ["10.0.0.2"] # own IP.
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: none
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

and here is the setup for es2 and es3:

http.host: 0.0.0.0
http.port: 9200
transport.host: _site_ # will bind to 10.0.0.3 (es2) resp. 10.0.0.4 (es4)
transport.port: 9300
node.name: "es2" # resp. "es3"
node.roles: [ data ]
cluster.name: elasticstack
discovery.seed_hosts: ["10.0.0.2"] # IP of es1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: none
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

You probably noticed, that I have disabled the SSL transport verification (verification_mode: none). I do so, since (currently) I do not distribute the transport.p12 across the cluster.

The issue is now, that I receive the following error (same error on all nodes, despite master or data role):

{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "Cluster state has not been recovered yet, cannot write to the [null] index"
            }
        ],
        "type": "authentication_processing_error",
        "reason": "failed to promote the auto-configured elastic password hash",
        "caused_by": {
            "type": "status_exception",
            "reason": "Cluster state has not been recovered yet, cannot write to the [null] index"
        }
    },
    "status": 503
}

I understand the discovery.seed_hosts mechanic, that it will automatically create a cluster. However, it seems that it does not do it.

How to automatically (via Terraform and user-data scripts) - without manual/human intervention - create a new Elasticsearch cluster, without having static secrets in the user-data script?