Unable to run set-up password after enabling x-pack security on a cluster

Elastic Stack Elasticsearch
1

Hello
I am using Elasticsearch version 7.6.2. on Ubuntu 18.04.
I set up an Elasticsearch cluster with 3 nodes and it works perfectly.

When I went back to set up secuirty on the cluster. I started by setting the following:

xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

Once this set Kibana ask for a username and password. Which I don't know.
I am unable to curl localhost:9200 without getting an authorization error.

I tried to run setup-passowrds command but that fails because it can't curl or post to the cluster.
I tried using the elasticsearch-users command to add a user and try to curl the cluster. But when I did that it only allowed me to curl localhost:9200 anything else and I would get the following errors:
On Kibana if I added the user I created to kibana.yml

{"statusCode":503,"error":"Service Unavailable","message":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];: [cluster_block_exception] blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}

on the terminal if I tried tocurl -u jim:changeme localhost:9200/_cat/nodes

"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Any thoughts from anyone? I am at my wits end with this issue.

2

I ended up deleting the whole cluster (which sucked because I had data in there) and reinstalling Elasticsearch.
I think because the node was once a cluster it refused to work when I attempted to turn it back to a single node. To avoid this when you delete Elasticsearch make sure you remove all the directories and files associate with it. For me that was /var/lib/elasticsearch and /usr/share/elasticsearch.
Once I reinstalled and turned my nodes to single node cluster everything worked fine, I am now in the process of restoring the cluster to what it was.

3

Did you see https://www.elastic.co/guide/en/elasticsearch/reference/7.6/configuring-security.html?

You can't just setup a single node in a cluster with security and have it rejoin, it needs to be all at once.

4

Hi Mark,

Yes, that is what I followed.
I re-installed all the nodes. Set them up as single nodes, then set up a ca on the master node and now i'm using that ca to sign the certificates and create the cluster.

5

Hi Mark,

I am now facing this same problem again but under different circumstances.

My master node is working fine and the encryption is set up. I set up my node to join and it worked fine and even joins the cluster. However, when I come to run the setup-passwords command it gives me the Failed to authenticate user 'elastic' against http://1.1.1.1:9200/_security/_authenticate?pretty.
I have set up users on this node yet.

Any suggestions to how I can fix this?
At the moment I am installing a clean install of Elasticsearch then setting up cluster, security, and encryption then starting the service. Once that's done I attempt to run the setup-passwords commands. Should I be running this command earlier?

6

I ended up figuring this issue out.

I didn't know once you join a node to a cluster it uses the master's users.

I curled on my slave node using the username and password for the master node and it worked fine.

1 Like
7

These details are cluster level, not node level.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.