Very First Elastic Install... Elastic fails to determine health

Technolust1 · March 29, 2024, 5:54pm

I'm new to ELK.. When I get to the step for checking if Elastic is running properly I get an error unable to authenticate. I realized I forgot to write down the password during the install... Now I'm trying to run elastic-reset-password -u elasticsearch however, I get the following error: ERROR: Failed to determine the health of the cluster. Unexpected http status [503], with exit code 65

I'm running 1 elastic server right now and planning to build a seperate kibanan, and separate logstash server after.

stephenb · March 30, 2024, 2:29pm

Hi @Technolust1 Welcome to the community.

You're going to have to provide a lot more information if you'd like help.

Basics we need to know:
What version did you install?
What Method did you use to install?
What operating system?
How are you starting the elasticsearch?
Did you look at the logs? We will need to see the logs.

Provide these and perhaps We can help

Technolust1 · April 1, 2024, 4:55am

Hey Stephen,

I guess some of the facts would be helpful, my apologies...

OS - Rocky 9.3

elastic verison - Version: 8.12.2, Build: rpm/48a287ab9497e852de30327444b0809e55d46466/2024-02-19T10:04:32.774273190Z, JVM: 21.0.2

installed downloading rpm file. also added an elasticrepo for update.

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

started Elasticsearch systemctl enable elasticsearch.service
systemcel start elasticsearch.service

tried to password reset from /usr/share/elasticsearch/bin/elasticsearch-reset-password

yes looked at the /var/log/elasticsearch/elasticsearch.log

Hope this helps. Let me know what else I can provide.

-Joe

stephenb · April 1, 2024, 5:18am

@Technolust1

Just telling us what you ran without the results we need to see command and results otherwise we are just guessing... More information better...help us help you

What does status show?

systemctl status elasticsearch

Show the command and the result... The command above you ran is incorrect... The is no user elasticsearch only elastic

And why are you trying to reset it ... When you installed the password was provided at the console... Reset is ok if needed.

What did they show...

Without detail we can not help... We are good but not that good.

Did you simply try to curl the endpoint? Even if the auth fails it will show valuable information.

curl -v -k -u elastic https://localhost:9200

Did you edit /etc/elasticsearch.yml or leave the defaults

Technolust1 · April 1, 2024, 6:43am

Hey Stephen,

I decided to rebuild the server from scratch in case it was something I did. I reset the password because I forgot to write it down when I installed. You were right, I used the wrong user, when I ran the reset tool on user elastic it worked.

Now:

[root@elastic bin]# ./elasticsearch -V
Version: 8.13.0, Build: rpm/09df99393193b2c53d92899662a8b8b3c55b45cd/2024-03-22T03:35:46.757803203Z, JVM: 22

[root@elastic yum.repos.d]# cd /usr/share/elasticsearch/bin/
[root@elastic bin]# ./elasticsearch -V
Version: 8.13.0, Build: rpm/09df99393193b2c53d92899662a8b8b3c55b45cd/2024-03-22T03:35:46.757803203Z, JVM: 22
[root@elastic bin]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-04-01 02:08:17 EDT; 8min ago
       Docs: https://www.elastic.co
   Main PID: 4427 (java)
      Tasks: 89 (limit: 48948)
     Memory: 4.2G
        CPU: 35.650s
     CGroup: /system.slice/elasticsearch.service
             ├─4427 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr>
             ├─4487 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative>
             └─4510 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Apr 01 02:08:06 elastic systemd[1]: Starting Elasticsearch...
Apr 01 02:08:08 elastic systemd-entrypoint[4427]: Apr 01, 2024 2:08:08 AM sun.util.locale.provider.LocaleProviderAdapter <cli>
Apr 01 02:08:08 elastic systemd-entrypoint[4427]: WARNING: COMPAT locale provider will be removed in a future release
Apr 01 02:08:17 elastic systemd[1]: Started Elasticsearch.

In the instructions it says to make sure elastic is running properly.

root@elastic elasticsearch]# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elasticsearch:$qvu8npyb+qIVQqvurGML https://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elasticsearch] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elasticsearch] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}[root@elastic elasticsearch]#

I did edit the /etc/elasticsearch/elasticsearch.yml file
Here is the changes I made:

cluster.name: elastic
network.host: 192.168.1.20
http.port: 9200
transport.host: 192.168.1.20

Here is running the curl on localhost:

[root@elastic bin]# curl -v -k -u elastic https://localhost:9200
Enter host password for user 'elastic':
*   Trying ::1:9200...
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=elastic
*  start date: Apr  1 05:59:12 2024 GMT
*  expire date: Apr  1 05:59:12 2026 GMT
*  issuer: CN=Elasticsearch security auto-configuration HTTP CA
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'elastic'
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/1.1
> Host: localhost:9200
> Authorization: Basic ZWxhc3RpYzpWRUlkM2ZxN3ZmUUp1QXlDeFZPUg==
> User-Agent: curl/7.76.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 526
< 
{
  "name" : "elastic",
  "cluster_name" : "elastic",
  "cluster_uuid" : "BwUiFOUjRteco5764evKIQ",
  "version" : {
    "number" : "8.13.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "09df99393193b2c53d92899662a8b8b3c55b45cd",
    "build_date" : "2024-03-22T03:35:46.757803203Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host localhost left intact

When I run the curl using the ip address:

[root@elastic bin]# curl -v -k -u elastic https://192.168.1.20:9200
Enter host password for user 'elastic':
*   Trying 192.168.1.20:9200...
* Connected to 192.168.1.20 (192.168.1.20) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=elastic
*  start date: Apr  1 05:59:12 2024 GMT
*  expire date: Apr  1 05:59:12 2026 GMT
*  issuer: CN=Elasticsearch security auto-configuration HTTP CA
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'elastic'
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/1.1
> Host: 192.168.1.20:9200
> Authorization: Basic ZWxhc3RpYzpyb290QGVsYXN0aWMgYmluXSMgY3VybCAtdiAtayAtdSBlbGFzdGljIGh0dHBzOi8vbG9jYWxob3N0OjkyMDA=
> User-Agent: curl/7.76.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< content-type: application/json
< content-length: 465
< 
* Connection #0 to host 192.168.1.20 left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}[root@elastic bin]# 
[root@elastic bin]#

/var/log/elasticsearch/elastic.log

root@elastic elasticsearch]# tail elastic.log 
[2024-04-01T02:08:18,863][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elastic] adding index lifecycle policy [.fleet-file-fromhost-meta-ilm-policy]
[2024-04-01T02:09:03,830][INFO ][o.e.x.s.a.RealmsAuthenticator] [elastic] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-01T02:20:49,973][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [elastic] users file [/etc/elasticsearch/users] changed. updating users...
[2024-04-01T02:20:49,975][INFO ][o.e.x.s.a.f.FileUserRolesStore] [elastic] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...
[2024-04-01T02:20:54,245][INFO ][o.e.x.s.s.SecurityIndexManager] [elastic] security index does not exist, creating [.security-7] with alias [.security]
[2024-04-01T02:20:54,273][INFO ][o.e.c.m.MetadataCreateIndexService] [elastic] [.security-7] creating index, cause [api], templates [], shards [1]/[0]
[2024-04-01T02:20:54,431][INFO ][o.e.c.r.a.AllocationService] [elastic] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-7][0]]])." previous.health="YELLOW" reason="shards started [[.security-7][0]]"
[2024-04-01T02:20:54,977][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [elastic] users file [/etc/elasticsearch/users] changed. updating users...
[2024-04-01T02:20:54,978][INFO ][o.e.x.s.a.f.FileUserRolesStore] [elastic] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...
[2024-04-01T02:27:28,895][INFO ][o.e.x.s.a.RealmsAuthenticator] [elastic] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[root@elastic elasticsearch]#

Hope this helps, let me know what else I can add.

-Joe

stephenb · April 1, 2024, 2:48pm

hi @Technolust1

You did not put the same password in the localhost attempt and IP attempt that is easy to tell as the

localhost

Authorization: Basic ZWxhc3RpYzpWRUlkM2ZxN3ZmUUp1QXlDeFZPUg==

ip

Authorization: Basic ZWxhc3RpYzpyb290QGVsYXN0aWMgYmluXSMgY3VybCAtdiAtayAtdSBlbGFzdGljIGh0dHBzOi8vbG9jYWxob3N0OjkyMDA=

The first one decodes properly ... the second does not in fact it decodes into a URL I think you pasted the wrong thing

Also

Technolust1:

In the instructions it says to make sure elastic is running properly.

root@elastic elasticsearch]# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elasticsearch:$qvu8npyb+qIVQqvurGML https://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elasticsearch] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elasticsearch] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}[root@elastic elasticsearch]#

you used elasticsearch again

finally, your http cert will need to be updated if you want to use the IP address with verification, you will need to create a new one and set it to be used by elasticsearch i.e. create it and then fix elasticsearch.yml

Your cert only has the hostname elastic you will need the IP if you want it to work properly with the CA.

* Server certificate:
*  subject: CN=elastic

I have some instructions on how to that here

github.com

bvader/howtos/blob/master/basic-security-elasticsearch-pkg/README.md


## Single Node Secured Elasticsearch + Kibana with Elastic generated self signed certs (updated)

#### NOTE / DISCLAIMER: **This configuration should only be used for Dev / POC purposes this is NOT suitable for production use.**

For Further Details Please Refer to the Official Documentation: [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) and [Kibana](https://www.elastic.co/guide/en/kibana/current/index.html)

### What we are doing?

This is simple / minimal quickstart to create a single Elasticsearch node and Kibana with basic authentication and SSL/TLS enabled (we will enable SSL for both HTTPS and Transport layer even though it is just a single node). This is a condensed / direct path to the Basic Security + HTTPs shown in the diagram [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html) and described [Configure security for the Elastic Stack](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html#security-basic-https-overview). We will then be able to bind the Elasticsearch and Kibana to the network so it can be safely reached from another system.  
**Do NOT bind your Elasticsearch node or cluster to the network unless you secure your cluster and Kibana FIRST!** 

![Elastic Securty Layers](https://www.elastic.co/guide/en/elasticsearch/reference/current/images/elastic-security-overview.png)

**Notes:** 
* This example is using Elastic Stack 7.15.1 and Ubuntu 20.04 LTS using a Deb Package, if you use another method such as the tar.gz you will need to adjust the paths. 
* We are colocating Elasticsearch and Kibana for POC / Dev purposes only.
* Many of these commands / directores require `root` access so either be prepared to `sudo` most of the commands or just do a `sudo -i` for the duration of the session... your choice...
* There are many important settings for Elasticsearch that are not in the example please review them [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html) before moving on from POC / Dev mode
* All the IP Adresses are just examples and are not to be taken literally.

This file has been truncated. show original

Technolust1 · April 1, 2024, 3:23pm

Well Dang, thanks for catching my stupidity! I think I may have got it...
As far as using elasticsearch again this is what I see using elastic user:

[root@elastic elasticsearch]# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:VEId3fq7vfQJuAyCxVOR https://localhost:9200
{
  "name" : "elastic",
  "cluster_name" : "elastic",
  "cluster_uuid" : "BwUiFOUjRteco5764evKIQ",
  "version" : {
    "number" : "8.13.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "09df99393193b2c53d92899662a8b8b3c55b45cd",
    "build_date" : "2024-03-22T03:35:46.757803203Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@elastic elasticsearch]# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:VEId3fq7vfQJuAyCxVOR https://192.168.1.20:9200
{
  "name" : "elastic",
  "cluster_name" : "elastic",
  "cluster_uuid" : "BwUiFOUjRteco5764evKIQ",
  "version" : {
    "number" : "8.13.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "09df99393193b2c53d92899662a8b8b3c55b45cd",
    "build_date" : "2024-03-22T03:35:46.757803203Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@elastic elasticsearch]#

Here is the verbose curl command using the ip address:

[root@elastic elasticsearch]# curl -v -k -u elastic https://192.168.1.20:9200
Enter host password for user 'elastic':
*   Trying 192.168.1.20:9200...
* Connected to 192.168.1.20 (192.168.1.20) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=elastic
*  start date: Apr  1 05:59:12 2024 GMT
*  expire date: Apr  1 05:59:12 2026 GMT
*  issuer: CN=Elasticsearch security auto-configuration HTTP CA
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'elastic'
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/1.1
> Host: 192.168.1.20:9200
> Authorization: Basic ZWxhc3RpYzpWRUlkM2ZxN3ZmUUp1QXlDeFZPUg==
> User-Agent: curl/7.76.1
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 526
<
{
  "name" : "elastic",
  "cluster_name" : "elastic",
  "cluster_uuid" : "BwUiFOUjRteco5764evKIQ",
  "version" : {
    "number" : "8.13.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "09df99393193b2c53d92899662a8b8b3c55b45cd",
    "build_date" : "2024-03-22T03:35:46.757803203Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host 192.168.1.20 left intact

As far as the cert goes, I will have to try that later this evening.
Let me know if this looks better?

I do have one more question: If I am using one VM for Elastic, a seperate VM for Kibana, and a third VM for Logstash, should I uncomment cluster.name: elastic in the elasticsearch.yml or is it only used if I'm actually clustering multiple elasticsearch servers?

Thoughts?

Thanks,

Joe

stephenb · April 1, 2024, 3:48pm

If you do not provide a cluster name, one will be provided for you free of charge. (i.e. a cluster always has a name)

It looks like your server name is also elastic , personally I try to keep things a little more identifiable...

Technolust1 · April 1, 2024, 4:11pm

Thanks Stephen, I will change the name. I just wasn't sure if I needed to actually use it because I'm not clustering elastic servers. I only used elastic as a generic one for community friendly naming...

leandrojmp · April 1, 2024, 4:51pm

In your case it is still a cluster, a single-node cluster.

Technolust1 · April 1, 2024, 5:31pm

Ah ok, make sense! Thanks for all of your help!

system · April 29, 2024, 5:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.