# Es 6.2.4 upgrade to es 6.3.0 but cannot join the cluster **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [September 13, 2019, 3:00am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345 "2019-09-13T03:00:00Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Bowen\_Liu](https://avatars.discourse-cdn.com/v4/letter/b/a4c791/32.png) [@Bowen\_Liu](https://discuss.elastic.co/u/Bowen_Liu)\ **Post date:** [September 13, 2019, 3:00am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345/1 "2019-09-13T03:00:00Z")
Hi, everyone. I encounter an upgrade problem and really don't know how to deal with it. The thing is that we need to upgrade es 6.2.4 to es 6.3.0, the cluster is build on some VM machines and the os is Ubuntu 16.04, there are master node, data node and client node in the cluster. We use security function and monitor function of x-pack in es 6.2.4 to According to the official guide, we can use rolling up upgrading to upgrade 6.2 to es 6.3. Here are my steps on master node: 1. First, disable the shard allocation and enable flush\_sync: 2. Download es 6.3.0 and cover the config file with old es 6.2.4's config. According to official guide line, x-pack is a build in module in es from 6.3 so I comment the x-pack config in the config file: config in 6.2.4 about x-pack: in es 6.3.0 I only keep one line: `xpack.security.enabled: true` Then I started es 6.3.0, there is no error message and the process seems to start up normally, but the node just cannot join the old cluster. It seems it even doesn't know the other master nodes' exist, and from the old cluster it also has nothing about this new node in log. First I think it may be because of the security thing, so I copy the config/certs dir of es 6.2 to es 6.3 and uncomment the following config: ``` xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/cert-node.p12 xpack.security.transport.ssl.truststore.path: certs/cert-node.p12 ``` When I started es 6.3 again, some error log was found: ``` Caused by: java.io.IOException: keystore password was incorrect ... Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... ``` It seems that es6.3 cannot use the es 6.2's certificate file. The certificate file is a global one and all nodes in es6.2 are the same. I started a new node with es6.2.4 use this cert is OK, the node join the cluster normally. As this is a live cluster, so we cannot just stop the cluster and start a new one. Now I really don't know what to do. Can some help? Thank you so much! ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [September 13, 2019, 7:01am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345/2 "2019-09-13T07:01:01Z")
> [@Bowen\_Liu](#): > > According to official guide line, x-pack is a build in module in es from 6.3 so I comment the x-pack config in the config file: The fact that x-pack is built in, doesn't mean that you don't need to configure it 🙂 You need to keep all the config you had in the previous elasticsearch.yml > [@Bowen\_Liu](#): > > Caused by: java.io.IOException: keystore password was incorrect ... Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. Please, _always_ share the full stack trace and not just small parts of this. This is really important for people trying to assist. If you don't give us all the information, we won't be able to help you. The error is self-explanatory: > Caused by: java.io.IOException: keystore password was incorrect From what I see, you have ```auto xpack.security.transport.ssl.keystore.path: certs/cert-node.p12 xpack.security.transport.ssl.truststore.path: certs/cert-node.p12 ``` One of these two ( or maybe both ) are password protected files, so you need to tell Elasticsearch how to decrypt them. I would guess that in your 6.2 configuration there is either a ```auto xpack.security.transport.ssl.keystore.password: xpack.security.transport.ssl.truststore.password: ``` section in elasticsearch.yml _or_ `xpack.security.transport.ssl.keystore.secure_password` and `xpack.security.transport.ssl.truststore.secure_password` are stored in the elasticsearch keystore ( you can check with `bin/elasticsearch-keystore list` in an old node. ---
**Author:** ![Bowen\_Liu](https://avatars.discourse-cdn.com/v4/letter/b/a4c791/32.png) [@Bowen\_Liu](https://discuss.elastic.co/u/Bowen_Liu)\ **Post date:** [September 14, 2019, 6:54am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345/3 "2019-09-14T06:54:17Z")
Thank you so much for your kindly reply. Actually, I don't find ``` xpack.security.transport.ssl.keystore.password: xpack.security.transport.ssl.truststore.password: ``` in the config file, and I do find a elasticsearch.keystore file in config dir. I think this is what you said ``xpack.security.transport.ssl.keystore.secure\_password`and`xpack.security.transport.ssl.truststore.secure\_password`are stored in the elasticsearch keystore` right? I ran `bin/elasticsearch-keystore list` in old node and the output is ``` keystore.seed xpack.security.transport.ssl.keystore.secure_password xpack.security.transport.ssl.truststore.secure_password ``` But when I ran same command in new node I only got `keystore.seed` I may know what happened but I still don't know **how to add the same key-store to the new node**. I copy the elasticsearch.keystore file to the new node but it seems no help. Should I ran `bin/elasticsearch-keystore create` to create a new one? I wonder if the new created one is the same with the old? ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [September 14, 2019, 7:16am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345/4 "2019-09-14T07:16:24Z")
You dont need to recreate the keystore, you just need to add the passwords to the keystore in the new node with ```auto bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password ``` but you need to know what the password is. ---
**Author:** ![Bowen\_Liu](https://avatars.discourse-cdn.com/v4/letter/b/a4c791/32.png) [@Bowen\_Liu](https://discuss.elastic.co/u/Bowen_Liu)\ **Post date:** [September 14, 2019, 7:36am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345/5 "2019-09-14T07:36:21Z")
Yes, I just find this in security guide line. But I don't know the password and the guy who build up this cluster has already gone. Is there another way to get the password or can I change the old password without stopping the cluster service? Many thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 12, 2019, 7:36am UTC](https://discuss.elastic.co/t/es-6-2-4-upgrade-to-es-6-3-0-but-cannot-join-the-cluster/199345/6 "2019-10-12T07:36:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.