Authentication issues while xpack security enabling

[TheVintik (Vitaly)]

Hello,

I have an issues while enabling security for ES. I have installed and configured several ES VMs. They already working as expected and connected to one ES cluster. After that, I decided to try to install and setup Kibana. There are also no any issues with Kibana installation. But, problems were started when I tried to setup authentication for Kibana...

As I read here: Configure security in Kibana | Kibana Guide [8.11] | Elastic - I need to Configuring security in Elasticsearch:
Configuring security in Elasticsearch | Elasticsearch Guide [7.5] | Elastic

So, point to point, from last link:

1. Verify that you are using a license that includes ...
As I understand - basic xpack security is free, so I can try to use it

2 Verify that the xpack.security.enabled...
Yes, enabled

3 If you plan to run Elasticsearch in a Federal Infox...
Skipped, no any plans

4 Configure Transport Layer Security ....
Yes, generated and prepared certificates

So... I just updated my ES config with these lines:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /.../certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /.../certs/elastic-certificates.p12

5 Restarting ES. ES was restarted successfully, and I started to get a message like:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

for any kind of request.

Ok, going to point 6 Set the passwords for all built-in users., I need to setup passwords for users with bin/elasticsearch-setup-passwords .... But... I got error:

Failed to determine the health of the cluster running at http://XXX:9200
Unexpected response code [503] from calling GET http://XXX:9200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]

So... I cant setup password, because I can not authentikate to my ES node... I tried to set xpack.security.enabled: false and run again ``bin/elasticsearch-setup-passwords ... but got error - I need to enable xpack security to setup password...

So, could anybody please advice me - what is wrong from my side? How I can setup password for system users?

[TheVintik (Vitaly)]

@TheVintik

Cause: master_not_discovered_exception

Ohh... my bad, just saw this error.. Fixed:

$ curl -XGET 'my_ip:9200/_cat/nodes?v'
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
my_ip            1          59  37    0.73    0.31     0.18 im        *      my_node

And, try to set passwords:

$ sudo bin/elasticsearch-setup-passwords auto

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.

The passwords will be randomly generated and printed to the console.

Please confirm that you would like to continue [y/N]y

Connection failure to: http://my_ip:9200/_security/user/apm_system/_password?pretty failed: Read timed out

ERROR: Failed to set password for user [apm_system].

$ telnet my_ip 9200

Trying my_ip...

Connected to my_ip.

Escape character is '^]'.

^] 

Any ideas?

[ikakavas (Ioannis Kakavas)]

Can you share the logs from elasticsearch around the time when you ran sudo bin/elasticsearch-setup-passwords auto ?

[TheVintik (Vitaly)]

Solved.
Looks like it was an issues in my cluster. I was confused because I can not see the cluster state with enabled xpack.security.enabled. So, looks like it was something like - "master node not eligible".

By the way... As I understand from documentation, I should setup passwords on each node using elasticsearch-setup-passwords. But, when I successfully run it on the first node, I successfully applied generated password on the second node. Interestin...

Aslo, it is very interesing - when I will add another node to the cluster - passwords will also applied to new node withoud running elasticsearch-setup-passwords utilite? As I understand - I am using File-based user authentication...