ES cluster break after install x-pack

Elastic Stack Elasticsearch
1

I have created ES cluster on MAC laptop with 1 client, 3 master and 3 data nodes.

$ cat multinode_es.sh
#!/bin/bash

### Client node

$HOME/Documents/ELK/ELASTIC/clientnode/elasticsearch-6.2.4/bin/elasticsearch &

### Master nodes
$HOME/Documents/ELK/ELASTIC/masternode/node1/elasticsearch-6.2.4/bin/elasticsearch &
$HOME/Documents/ELK/ELASTIC/masternode/node2/elasticsearch-6.2.4/bin/elasticsearch &
$HOME/Documents/ELK/ELASTIC/masternode/node3/elasticsearch-6.2.4/bin/elasticsearch &

### Data nodes

$HOME/Documents/ELK/ELASTIC/datanode/node1/elasticsearch-6.2.4/bin/elasticsearch &
$HOME/Documents/ELK/ELASTIC/datanode/node2/elasticsearch-6.2.4/bin/elasticsearch &
$HOME/Documents/ELK/ELASTIC/datanode/node3/elasticsearch-6.2.4/bin/elasticsearch &

Everything was working fine with this setup before installing the x-pack, i have installed the x-pack on all master nodes using elasticsearch-plugin command.

Killed all the ES related processes and started them again but after that it's getting failed with the following error:

2

https://paste.fedoraproject.org/paste/0h~v7KkZhXZdj~XFkTtCwQ

Since cluster is not started properly hence x-pack command to generate the password is also not working.

$ bin/x-pack/setup-passwords auto

Connection failure to: http://127.0.0.1:9200/_xpack/security/_authenticate?pretty failed: Connection refused (Connection refused)

ERROR: Failed to connect to elasticsearch at http://127.0.0.1:9200/_xpack/security/_authenticate?pretty. Is the URL correct and elasticsearch running?

Strangely my head plugin which is installed as chrome extension is asking for the password to connect with ES before installing x-pack it was not asking for password.

Why ES is not starting properly and which password I am supposed to put in head so that it can fetch the ES information?

3

You need to install X-Pack on all your nodes, not only on the master nodes.

4

Thanks it helped me to start the cluster but while resetting the password I am getting this error.

$ bin/x-pack/setup-passwords interactive

Failed to authenticate user 'elastic' against http://127.0.0.1:9200/_xpack/security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /Users/viaggarw/Documents/ELK/ELASTIC/datanode/node3/elasticsearch-6.2.4/config/elasticsearch.keystore

ERROR: Failed to verify bootstrap password
HAM-VIAGGARW-02:elasticsearch-6.2.4 viaggarw$ cat /Users/viaggarw/Documents/ELK/ELASTIC/datanode/node3/elasticsearch-6.2.4/config/elasticsearch.keystore

I am not able to read the content of mentioned file.

I have tried to login using elastic/changeme credentials as indicated in other discussions but that didn't work

$ curl -u elastic -XGET 'http://localhost:9200/_cat/health?v'
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
5

Because by installing X-Pack with the defaults you have secured the REST API that your plugin previously spoke to unsecured.

This is by design, it's an encrypted Java keystore meant to protect secrets.

You can check which keys (but not their passwords) the keystore contains by entering

bin/elasticsearch-keystore list

in the path for that node (note it is not the x-pack directory within bin/.

As this is on your laptop, your easiest option is probably to stop your cluster, remove X-Pack from all 7 nodes with

bin/elasticsearch-plugin remove x-pack

double check a keystore is not present in any config path, then reinstall to all nodes with

bin/elasticsearch-plugin install x-pack

then start the cluster back up and run

bin/x-pack/setup-passwords interactive

1 Like
6

The documentation for bootstrap passwords when installing X-Pack can be found here.

7

Thanks for your reply.

Removed the xpack using --purge option.

$ sh multinode_remove_xpack.sh
-> removing [x-pack]...
-> removing [x-pack]...
-> removing [x-pack]...
-> removing [x-pack]...
-> removing [x-pack]...
-> removing [x-pack]...
-> removing [x-pack]...

This is the keystore present in keystore. I deleted from one of the node, should I delete it from all nodes?

$ sh multinode_keystore.sh
keystore.seed
keystore.seed
keystore.seed
keystore.seed
keystore.seed
keystore.seed
8

Yes I would say that is your best option, you want a clean install of X-Pack to all nodes, not one where they pick up an existing keystore and lead you into the same problem.

9

I cleared the keystores after the uninstallation of x-pack. Started the ES nodes and verified that I am able to access the setup without credentials.

Again did the installation of x-pack, stopped/started the ES services.

it's asking for credentials.

10

You need to run

bin/x-pack/setup-passwords interactive

after you start the cluster to set the credentials for the reserved users..

11

Thanks I was able to change this time. but it should have some default password for the configured users because as soon as you install x-pack it stated asking for the password. changeme doesn't work while trying to authenticating the api.

$ bin/x-pack/setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,kibana,logstash_system.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [elastic]
12

Hi. No it shouldn't. Removing the default changeme password was a deliberate change in 6.0 with the user's security in mind while at the same time preserving the ease of installation. You can read more about it in this blog post

If by "it" you refer to the elasticsearh head chrome extention, I won't comment on a 3rd party software but our documentation on running setup-passwords right after you start your cluster for the first time after installing XPack are, I believe, clear.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.