Can't start ES 7.3.0 with x-pack security enabled

[ludaca]

Hello,
I'm trying to enable xpack.security in ES 7.3.0 fresh install on k8s.
X-pack license type: basic

My elasticsearch.yml
</> cluster.name: "es7-sec"
network.host: 0.0.0.0
path.logs: /var/log
discovery.seed_hosts:
- es-master-0
- es-master-1
.........
- es-data1-2
cluster.initial_master_nodes:
- es-master-0
- es-master-1
- es-master-2
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
</>

elastic-certificates.p12 generated using docker-compose with ES image 7.3.0 by command":
bin/elasticsearch-certutil cert --silent --pass xxxxx -out elastic-certificates.p12
and mounted on all nodes in ES cluster under /usr/share/elasticsearch/config with permissions:
600 elasticsearch:root elastic-certificates.p12

Containers crashed with such messages in ES log:
</> "Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]",
"Caused by: java.lang.reflect.InvocationTargetException",
"Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory",
"Caused by: java.io.IOException: keystore password was incorrect",
"Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption."
</>

Any help will be appreciated.
Thanks in advance
Luda

[ikakavas (Ioannis Kakavas)]

The problem is that you are setting a password for your PKCS#12 file by passing --pass parameter, but then you're not providing that password to Elasticsearch for it to be able to decrypt and read your elastic-certificates.p12

If you want/need to use a password protected keystore/truststore, then you need to set also

xpack.security.transport.ssl.keystore.password: 
xpack.security.transport.ssl.truststore.password: 

so that Elasticsearch can read your elastic-certificates.p12

[ludaca]

Thank you, the masters started now, but in ES logs there is:
</> "message": "[xpack.security.transport.ssl.truststore.password] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." } </>
Is there a possibility to create p12 cert without password?

[crickes (Steve Crickett)]

The parameter names has been changed.
Its now called: xpack.security.transport.ssl.truststore.secure_password

[ikakavas (Ioannis Kakavas)]

@ludaca

Is there a possibility to create p12 cert without password?

Unfortunately you can't create a PKCS12 store without setting a password, you can only set an empty password, see the documentation for --pass parameter in elasticsearch-certutil | Elasticsearch Guide [8.11] | Elastic.

Possible solution 1

[Editing this to keep everything in place for future reference]
As @TimV mentioned in the post below correcting me:

@TimV

We assume that the password is blank if it is not specified. Simply create a PKCS#12 file with a blank password, and leave the .password (and .secure_password ) unset.

Possible solution 2

As @crickes mentioned, you can either use the equivalent secure settings:

xpack.security.transport.ssl.keystore.secure_password:
xpack.security.transport.ssl.truststore.secure_password:

that need to be set in the secure settings

To be precise, these have not been changed but password has been deprecated in favor of secure_password.

Possible solution 3

If you can't/don't want to use secure settings, then you alternatively use elasticsearch-certutil to create a PEM formatted key and certificate that do not need to be password protected, using the --pem parameter

i.e.

bin/elasticsearch-certutil cert --silent --pem -out elastic-certificates-pem.zip

and unzipping this you will get 3 files

• ca/ca.crt
• instance/instance.crt
• instance/instance.key

Then you can configure your nodes with

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.key: instance.key
xpack.security.transport.ssl.certificate:instance.crt 
xpack.security.transport.ssl.certificate_authorities: [ "ca.crt" ] 

[TimV (Tim Vernum)]

@ikakavas

If you want/need to use a password protected keystore/truststore, then you need to set also

xpack.security.transport.ssl.keystore.password: 
xpack.security.transport.ssl.truststore.password: 

This should not be necessary. We assume that the password is blank if it is not specified. Simply create a PKCS#12 file with a blank password, and leave the .password (and .secure_password) unset.

[ludaca]

Thanks so much to everyone for the detailed answers!