Elasticsearch ignores config file /etc/elasticsearch/elasticsearch.yml

Elastic Stack Elasticsearch
1

Hello.
I have a single node elasticsearch version 8.13.4. It was installed using the repository on Ubuntu version 22.04.
Just yesterday, I decided to update the xpack.security.transport.ssl ​​certificates. I regenerated the certificates using the new CA. I made changes to the file /etc/elasticsearch/elasticsearch.yml and restarted the service.
Elasticsearch failed to start due to an error:

[2022-10-12T13:54:57,098][ERROR][o.e.b.Elasticsearch      ] [es-1] fatal exception while booting Elasticsearch
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/transport.p12] - this is usually caused by an incorrect password; (no password was provided)
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:605) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:463) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:312) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$15(Node.java:696) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:236) ~[elasticsearch-8.4.3.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:710) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.3.jar:?]
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/transport.p12] - this is usually caused by an incorrect password; (no password was provided)
        at org.elasticsearch.common.ssl.SslFileUtil.ioException(SslFileUtil.java:56) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:98) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        ... 23 more
Caused by: java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2158) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:226) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1503) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        ... 23 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2158) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:226) ~[?:?]
        at java.security.KeyStore.load(KeyStore.java:1503) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:473) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:603) ~[?:?]
        ... 23 more

The problem is that the file /etc/elasticsearch/certs/transport.p12 is not listed anywhere in my current configuration. It looks like elasticsearch is ignoring the new elasticsearch.yml file. It might be a stuck configuration cache. But I can't find a way to reset it.
My elasticsearch.yml:

grep -v '^$\|^#' /etc/elasticsearch/elasticsearch.yml
cluster.name: clustername
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/elastic-certificates.p12
  # The password for the keystore
  # keystore.secure_password: "password"
  # The password for the key in the keystore. The default is the keystore password
  # keystore.secure_key_password: "password"
  truststore.path: certs/elastic-certificates.p12
cluster.initial_master_nodes: ["es-1"]
http.host: 0.0.0.0
transport.host: 10.10.1.101
2

Hi @Dmitry_Stallion Welcome to the community.

Elasticsearch is acting like it can not find the elasticsearch.yml files and is relying on the defaults. Did you accidently change the permissions or owner?

Please run the following and share the output
cd /etc/elasticsearch
ls -la

perhaps also try

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
3

OMG! I am stupid. The log file was wrong. The date of log 2022-10-12.
The real problem with /etc/elasticsearch/certs/http.p12 file. I found it in different log file.

Somebody, close the ticket please)

1 Like