X-Pack security TLS/SSL not working with own certificates

Elastic Stack Elasticsearch
1

I recently configured my elastic cluster on ssl, with the ssl-tool from elastic this was working fine. Now I want to change the keys with keys from our own CA. And here I got stuck on the error below.

Allthough the Error states the password is incorrect, I know its correct. And when I remove the password from the pfx, the errors stays the same. I suspect my config is at fault, but I can't find where and how.

Config:

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-ota.pfx
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.keystore.password: <>
xpack.security.transport.ssl.truststore.path: elastic-ota.pfx
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.truststore.password: <>

ERROR:

Summary

[2018-10-10T09:38:32,821][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.node.Node.(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.3.0.jar:6.3.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.3.0.jar:6.3.0]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0]
... 15 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:60) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_181]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:79) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:134) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0]
... 15 more
Caused by: java.io.IOException: keystore password was incorrect
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_181]
at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:265) ~[?:?]
at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:256) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_181]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:79) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:134) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0]
... 15 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

2

I think you may have hit an issue with the JDK's support for PKCS12 files. If an alias is missing for a certificate, this error can happen. Do you have access to openssl? If so you can try the following:

openssl pkcs12 -in  elastic-ota.pfx  -nocerts -out key.pem
openssl pkcs12 -in elastic-ota.pfx -clcerts -nokeys -out cert.pem
openssl pkcs12 -in elastic-ota.pfx -cacerts -nokeys -out ca-cert.pem
openssl pkcs12 -export -out elastic-ota-with-alias.pfx -inkey key.pem -in cert.pem -certfile ca-cert.pem -name "alias"
1 Like
3

I changed the alias name with your openssl input. But now stranger this are happening. Elasticsearch doesn't start any more and no logs are writen to de elastic logs. The thing I could find in de messages log was:

Blockquote
Oct 15 10:31:35 slao25469 java[6357]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: org.elasticsearch.cli.UserException: unable to create temporary keystore at [/etc/elasticsearch/elasticsearch.keystore.tmp], please check filesystem permissions
Oct 15 10:31:35 slao25469 java[6357]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp
Oct 15 10:31:35 slao25469 java[6357]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Oct 15 10:31:35 slao25469 java[6357]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Oct 15 10:31:35 slao25469 java[6357]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Oct 15 10:31:35 slao25469 java[6357]: at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
Oct 15 10:31:35 slao25469 java[6357]: at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
Oct 15 10:31:35 slao25469 java[6357]: at java.nio.file.Files.newOutputStream(Files.java:216)
Oct 15 10:31:35 slao25469 java[6357]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
Oct 15 10:31:35 slao25469 java[6357]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
Oct 15 10:31:35 slao25469 java[6357]: at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:458)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:234)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.cli.Command.main(Command.java:90)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Oct 15 10:31:35 slao25469 java[6357]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Oct 15 10:31:35 slao25469 java[6357]: Refer to the log for complete error details.
Oct 15 10:31:35 slao25469 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 15 10:31:35 slao25469 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 15 10:31:35 slao25469 systemd[1]: elasticsearch.service failed.

I changed the rights on the config folder, so elastic is able to right the temp key. Then I end up with this error, no other changes have been made to the installation.

Blockquote
Oct 15 10:55:19 slao25469 systemd[1]: Starting Elasticsearch...
Oct 15 10:55:20 slao25469 elasticsearch[12282]: /usr/share/elasticsearch/bin/elasticsearch-env: line 78: cd: /etc/elasticsearch: Permission denied
Oct 15 10:55:20 slao25469 elasticsearch[12282]: Exception in thread "main" java.nio.file.NoSuchFileException: /usr/share/elasticsearch/jvm.options
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at java.nio.file.Files.newByteChannel(Files.java:361)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at java.nio.file.Files.newByteChannel(Files.java:407)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at java.nio.file.Files.newInputStream(Files.java:152)
Oct 15 10:55:20 slao25469 elasticsearch[12282]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:58)
Oct 15 10:55:20 slao25469 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 15 10:55:20 slao25469 systemd[1]: Unit elasticsearch.service entered failed state.

4

Most likely ES_PATH_CONF is set to a directory without execute permission.

You seem to be hitting a lot of filesystem permission errors. The ES_PATH_HOME (the instalation) dir should be a directory that should be writable and executable under the user running the ES process.

5

I checked the permissions and indeed they had no execute rights. Seems that elastic also checks all unused folders in the conf_path. After correcting that everything is working again.

I checked a clean install of elastic and with the standard installation of the RPM and the rights on the ES_CONF_PATH are default set:
drwxr-s---. 3 root elasticsearch 4096 Sep 7 10:38 elasticsearch

Thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.