Secure_Password in ES Keystore for HTTP Exporters

Dietmar · June 19, 2018, 12:23pm

When I set the passphrase for the parameter xpack.monitoring.exporters.admincluster.ssl.keystore.secure_password in ES-Keestore, an error occurs when starting ES: java.lang.IllegalStateException: password has been cleared.

Is anyone able to help me?

Viel Grüße
Dietmar

ikakavas · June 19, 2018, 3:01pm

Hi,

Please tell us which version of Elasticsearch you are running and paste here a portion of your logs around where the exception is thrown

Dietmar · June 19, 2018, 7:23pm

Hallo,

the Version is 6.1.3.

Here the log snipped :
[2018-06-19T13:22:10,625][INFO ][o.e.n.Node ] [ls01130y] initialized
[2018-06-19T13:22:10,625][INFO ][o.e.n.Node ] [ls01130y] starting ...
[2018-06-19T13:22:10,648][ERROR][o.e.b.Bootstrap ] [ls01130y] Exception
java.lang.IllegalStateException: password has been cleared
at java.security.KeyStore$PasswordProtection.getPassword(KeyStore.java:347) ~[?:1.8.0_152]
at sun.security.pkcs12.PKCS12KeyStore.engineGetEntry(PKCS12KeyStore.java:1304) ~[?:?]
at java.security.KeyStore.getEntry(KeyStore.java:1521) ~[?:1.8.0_152]
at org.elasticsearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:346) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:152) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:142) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.SecureSetting.get(SecureSetting.java:93) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.xpack.ssl.SSLConfiguration.createKeyConfig(SSLConfiguration.java:217) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLConfiguration.(SSLConfiguration.java:75) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.sslConfiguration(SSLService.java:334) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.sslIOSessionStrategy(SSLService.java:137) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.configureSecurity(HttpExporter.java:429) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.createRestClient(HttpExporter.java:262) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.(HttpExporter.java:196) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.(HttpExporter.java:183) ~[?:?]
at org.elasticsearch.xpack.monitoring.Monitoring.lambda$createComponents$1(Monitoring.java:157) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.initExporters(Exporters.java:165) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.doStart(Exporters.java:89) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:66) ~[elasticsearch-6.1.3.jar:6.1.3]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_152]
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) ~[?:1.8.0_152]
at org.elasticsearch.node.Node.start(Node.java:588) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:261) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:331) [elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.1.3.jar:6.1.3]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.1.3.jar:6.1.3]
[2018-06-19T13:22:10,657][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [ls01130y] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: password has been cleared
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.1.3.jar:6.1.3]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.1.3.jar:6.1.3]
Caused by: java.lang.IllegalStateException: password has been cleared
at java.security.KeyStore$PasswordProtection.getPassword(KeyStore.java:347) ~[?:1.8.0_152]
at sun.security.pkcs12.PKCS12KeyStore.engineGetEntry(PKCS12KeyStore.java:1304) ~[?:?]
at java.security.KeyStore.getEntry(KeyStore.java:1521) ~[?:1.8.0_152]
at org.elasticsearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:346) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:152) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:142) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.settings.SecureSetting.get(SecureSetting.java:93) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.xpack.ssl.SSLConfiguration.createKeyConfig(SSLConfiguration.java:217) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLConfiguration.(SSLConfiguration.java:75) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.sslConfiguration(SSLService.java:334) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.sslIOSessionStrategy(SSLService.java:137) ~[?

Viele Grüße
Dietmar

ikakavas · June 20, 2018, 7:56am

Was the command to store the password in the elasticsearch keystore successful ? It does look like there is no entry for that key in the keystore.

Does running
bin/elasticsearch-keystore list

print out xpack.monitoring.exporters.admincluster.ssl.keystore.secure_password among others ?

Dietmar · June 20, 2018, 8:19am

Yes, the command was successful and the output of bin/elasticsearch-keystore list shows xpack.monitoring.exporters.admincluster.ssl.keystore.secure_password.

Viele Grüße

Dietmar

TimV · June 20, 2018, 12:16pm

Sorry, this is a bug in all current versions of X-Pack.

The only workaround at this time is to enter the password using the non secure version.
This will be fixed in an upcoming release.

Dietmar · June 21, 2018, 5:32am

Thank you for the answers.

We solved this problem by using the same keystore for ES-Server and Monitoring-Client.
For this we have only set the global xpack.ssl Parameter. See below :

    xpack:
      monitoring.exporters:
        admincluster:
          type: http
          host:
            - ${ES_ADMINCLUSTER_NODE1}
            - ${ES_ADMINCLUSTER_NODE2}
      ssl:
        keystore:
          path: ${ES_PATH_CONF}/x-pack/${HOSTNAME}.p12
          type: PKCS12
        certificate_authorities: ${ES_PATH_CONF}/x-pack/Root_CA_01.cer
        supported_protocols: TLSv1.2
        client_authentication: required
        verification_mode: full
      security:
.
.
.

The password for the pkcs12-Keystore is then stored in the ES-keystore with Parameter xpack.ssl.keystore.secure_password

Viele Grüße

Dietmar

Topic		Replies	Views
Unable to start elasticsearch after add keystore on RHEL7 Elastic Security	5	1307	November 4, 2022
Unable to store monitoring credentials in ES Keystore Elasticsearch	1	637	July 2, 2018
Elasticsearch.keystore Elasticsearch	8	3102	November 28, 2018
Keystore for Monitoring Cluster settings Elasticsearch	1	313	July 12, 2018
Keystore.password deprecated Elasticsearch elastic-stack-security	8	2198	December 22, 2022

Secure_Password in ES Keystore for HTTP Exporters

Related topics