Unable to run set-password in 6.x

[stephenpatten]

Hello,

I'm setting up a 6.1.2 windows cluster and have a gold license. This is the first machine in the cluster so I have generated a CA cert with a password and placed it in the config directory.

relevant keys from the elasticsearch.yml file:

xpack.ssl.keystore.path: elastic-stack-test-ca.p12
xpack.ssl.truststore.path: elastic-stack-test-ca.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true

I have run through the docs and am currenlty trying to complete step 7

Unfortunately this command "Elasticsearch\6.1.2\bin\x-pack\setup-passwords auto" is throwing an exception.

Exception:

Exception in thread "main" ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded];
at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61)
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:408)
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:444)
at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:87)
at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.postURL(CommandLineHttpClient.java:91)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:278)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:127)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:75)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:105)
Caused by: java.io.IOException: keystore password was incorrect
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.elasticsearch.xpack.ssl.CertUtils.readKeyStore(CertUtils.java:230)
at org.elasticsearch.xpack.ssl.CertUtils.trustManager(CertUtils.java:221)
at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:59)

I don't see anywhere in the Set Passwords Command where you pass the password for the keystore.

Hoping someone can shed some light on what I might be doing wrong, or my next steps.

UPDATE

I've taken a step back and started at the basics by reading this blog post.

I did start out with the latest bits, 6.1.2 on a windows enterprise 10 machine, but aside from that the steps were followed to the letter. Note that I am using cmder with terminals, node1 term, node2 term, and issuing the command in term3 (C:\tmp\cert_blog\elasticsearch-6.1.2)

Both nodes came up and the logs looked good, but when trying to issue this command (again in term3):

C:\tmp\cert_blog\elasticsearch-6.1.2
λ bin\x-pack\setup-passwords auto -u "https://node1.local:9200"

the response is

SSL connection to https://node1.local:9200/_xpack/security/_authenticate?pretty failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

ERROR: Failed to establish SSL connection to elasticsearch at https://node1.local:9200/_xpack/security/_authenticate?pretty.

here are the nodes ymal files:

node.name: node1
network.host: node1.local
xpack.ssl.key: certs\node1.key
xpack.ssl.certificate: certs\node1.crt
xpack.ssl.certificate_authorities: certs\ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
discovery.zen.ping.unicast.hosts: [ 'node1.local', 'node2.local']
node.max_local_storage_nodes: 2

node.name: node2
network.host: node2.local
xpack.ssl.key: certs\node2.key
xpack.ssl.certificate: certs\node2.crt
xpack.ssl.certificate_authorities: certs\ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
discovery.zen.ping.unicast.hosts: [ 'node1.local', 'node2.local']
node.max_local_storage_nodes: 2

paths to the certs:

C:\tmp\cert_blog\elasticsearch-6.1.2\config\certs
ca.crt
node1.crt
node1.key

C:\tmp\cert_blog\elasticsearch-6.1.2\config2\certs
ca.crt
node2.crt
node2.key

I feel much more confident about where I am in this process versus the first one at the beginning of this post. Do I need to install the ca into windows so that cmder can issue the commands?

Thank you,
Stephen

[bhavyarm]

Hi,

Can you please raise this question to your elastic support? You have a license and somebody will get to you ASAP it's feasible.

Thanks,
Bhavya

[ikakavas (Ioannis Kakavas)]

Contacting support is the best way forward for your case, and I hope this is already resolved.

For what is worth, from your description, it looks as though you haven't set the certificate password in the Elasticsearch internal keystore as described here in step 2

[TimV (Tim Vernum)]

Support is definitely the best option here, but in order to make as much information available online here's a few pointers:

• Generally I would recommend running setup-passwords before configuring TLS. You don't have to do that, but when you're starting out for the very first time, I think that the most straight forward path is the order below. It does require starting and stopping you node(s) multiple times, and changing configuration as you go, so it's not suitable for every environment (which is why it's not the instructions in the docs) but if you want a path where you do 1 step, verify it worked, and then do the next step, this is the way to go:

  1. Install a single ES node.
  2. Install X-Pack on that node.
  3. Run setup-passwords on that node.
  4. Add additional nodes (ES + X-Pack).
  5. Enable TLS for node transport.
  6. Enable TLS for HTTP.

• We have a trouble shooting guide for setup-passwords. It needs a little bit more curating, but it does describe your most recent issue: https://www.elastic.co/guide/en/x-pack/current/trb-security-setup.html

• The challenge with setup-passwords is that it acts as a client of the ES HTTP API, but Elasticsearch is not normally a consumer of its own API. So you can have a perfectly configured ES cluster, but it doesn't know how to talk to itself (because it normally doesn't need to). That usually becomes a problem when SSL/TLS is involved because the steps you need to do to make Elasticsearch work as an HTTP server with TLS are not the same as what you need to make a client talk to Elasticsearch over HTTPS.

• In this case, I think you just want to add the following to the elasticsearch.yml for each node.

  xpack.security.http.ssl.certificate_authorities: certs\ca.crt
  

[stephenpatten]

Thanks Tim!

The error is #2 from the troubleshooting guide and adding in 'xpack.security.http.ssl.certificate_authorities: certs\ca.crt' did not solve the issue.

Again, thanks for your help and now I don't feel that crazy.

I tend to like the approach of doing the certs first as the blog outlined, it seemed to make more sense creating them first instead of backing into them after the clusters have been built. Pros and Cons to both approaches I guess. By the time I get this running I'll look back at these days and think, oh that was easy.... I hope

[TimV (Tim Vernum)]

Another possibility is that setup-passwords is picking up the wrong configuration file.

Can you try adding --verbose to the end of the command line, and see whether the increased output provides any clues?

[stephenpatten]

Here is the verbose output:

C:\tmp\cert_blog\elasticsearch-6.1.2
λ bin\x-pack\setup-passwords auto -u "https://node1.local:9200" --verbose
Running with configuration path: C:\ProgramData\Elastic\Elasticsearch\config

Testing if bootstrap password is valid for https://node1.local:9200/_xpack/security/_authenticate?pretty

SSL connection to https://node1.local:9200/_xpack/security/_authenticate?pretty failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alerts.getSSLException(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
at java.base/sun.security.ssl.Handshaker.fatalSE(Unknown Source)
at java.base/sun.security.ssl.Handshaker.fatalSE(Unknown Source)
at java.base/sun.security.ssl.ClientHandshaker.checkServerCerts(Unknown Source)
at java.base/sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
at java.base/sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
at java.base/sun.security.ssl.Handshaker.processLoop(Unknown Source)
at java.base/sun.security.ssl.Handshaker.processRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.processInputRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
at org.elasticsearch.xpack.common.socket.SocketAccess.lambda$doPrivileged$0(SocketAccess.java:54)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:53)
at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.postURL(CommandLineHttpClient.java:110)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:278)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:127)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:75)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:105)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
at java.base/sun.security.validator.Validator.validate(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568)
... 26 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
... 33 more

ERROR: Failed to establish SSL connection to elasticsearch at https://node1.local:9200/_xpack/security/_authenticate?pretty.

Can I swear in the forum? This is a by-product of having run the windows msi for one of my other early spikes...Let me uninstall the msi and cleanup after myself.

After the uninstall of the 6.1.2 msi here is the new output:

C:\tmp\cert_blog\elasticsearch-6.1.2
λ bin\x-pack\setup-passwords auto -u "https://node1.local:9200" --verbose
Running with configuration path: C:\ProgramData\Elastic\Elasticsearch\config
ERROR: Elasticsearch keystore file is missing [C:\ProgramData\Elastic\Elasticsearch\config\elasticsearch.keystore]

Also, a cursory glance at the environment variables and I don't see the ES_HOME or ES_PATH_CONF. Going to reboot, and will post my next sets of findings

-Stephen

[TimV (Tim Vernum)]

If it helps, the only reason I thought to mention it was that I went to check whether --verbose provided any useful output for this scenario, and accidentally ran it in a terminal window that had ES_PATH_CONF set to the wrong place

It's a pretty common problem (which is why I added that as the first line of the verbose output, but maybe we need to print it out all the time)

[stephenpatten]

Here is the new output after a reboot:

C:\tmp\cert_blog\elasticsearch-6.1.2
λ bin\x-pack\setup-passwords auto -u "https://node1.local:9200" --verbose
Running with configuration path: C:\tmp\cert_blog\elasticsearch-6.1.2\config
ERROR: Elasticsearch keystore file is missing [C:\tmp\cert_blog\elasticsearch-6.1.2\config\elasticsearch.keystore]

Environment variables:
ES_HOME = "C:\tmp\cert_blog\elasticsearch-6.1.2"
ES_PATH_CONF = "C:\tmp\cert_blog\elasticsearch-6.1.2\config"

[TimV (Tim Vernum)]

@stephenpatten

ERROR: Elasticsearch keystore file is missing [C:\tmp\cert_blog\elasticsearch-6.1.2\config\elasticsearch.keystore]

That shouldn't happen. Did you manually create the config directory? Or copy one from another ES installation?

The keystore file should be created automatically when you install X-Pack.
The password functionality relies on the keystore existing, as it contains the secret initial password that is used to authenticate the setup command.

You can create a new one (which will automatically have the bootstrap secret) with bin/elasticsearch-keystore create. You'll need to restart ES after you do that (it doesn't automatically reload the keystore).

[stephenpatten]

Not sure about the original node1 config, but the second one for node2 was created by the hand. I believe issuing this command on windows from the step Elasticsearch TLS setup in the blog post might have potentially been the issue, or myself, take your pick.

Great news!

Running with configuration path: C:\tmp\cert_blog\elasticsearch-6.1.2\config

Testing if bootstrap password is valid for https://node1.local:9200/_xpack/security/_authenticate?pretty
{
"username" : "elastic",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}

Initiating the setup of passwords for reserved users elastic,kibana,logstash_system.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y

Trying user password change call https://node1.local:9200/_xpack/security/user/kibana/_password?pretty
{ }

Changed password for user kibana
PASSWORD kibana = changeme

Trying user password change call https://node1.local:9200/_xpack/security/user/logstash_system/_password?pretty
{ }

Changed password for user logstash_system
PASSWORD logstash_system = changeme

Trying user password change call https://node1.local:9200/_xpack/security/user/elastic/_password?pretty
{ }

Changed password for user elastic
PASSWORD elastic = changeme

Now that's what I'm talking about, I been here before so this looks GREAT. I'd like to thank you for digging me out Tim, much appreciated.