ElasticSearch does not start and not even cluster

Elastic Stack Elasticsearch
1

I'm trying to create an Elasticsearch cluster with 3 nodes, each node being eligible as a master, as stated in this doc.

This cluster will be used by the end user only on our local network through an nginx proxy that will point his port 9200 to port 9200 of one of the three nodes.

For that, I configured each node to use network.host with the fixed local IP of each machine.

It turns out that even locally when running the command curl -XGET '192.168.0.1:9200/_cluster/health?pretty' I just get the response

curl: (52) Empty reply from server.

What could I have done wrong?

These are the steps I did for configuration and how my configuration files turned out:

1 - I installed ES using Debian packages and kept certificates as it generated, and user as well.
2 - After installing, run the commands below:

sudo mkdir -p /etc/systemd/system/elasticsearch.service.d/
sudo nano /etc/systemd/system/elasticsearch.service.d/override.conf

...and added the line...

[Service]
LimitMEMLOCK=infinity

3 - I modified jvm.options to use 4GB of RAM, as the server has 8GB of RAM

sudo nano /etc/elasticsearch/jvm.options

-Xms4g
-Xmx4g

4 - I edited the elasticsearch.yml file with the following settings:

cluster.name: my-elk-cluster
node.name: master-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.0.1
discovery.seed_hosts: ["192.168.0.1", "192.168.0.2", "192.168.0.3"]
cluster.initial_master_nodes: ["master-01", "master-02", "master-03"]

# Below automatically generated by Elastic on install
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

5 - And after that, I just configured Elasticsearch to start with the server:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

The command sudo journalctl --unit elasticsearch does not show any errors.

2

What does systemctl status elasticsearch return on your nodes?

Also, you have security enabled, you need to both use https and authenticate with the password created during the installation.

3

Hi @leandrojmp,
This is the return

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─override.conf
     Active: active (running) since Tue 2023-06-27 14:57:11 -03; 44min ago
       Docs: https://www.elastic.co
   Main PID: 461 (java)
      Tasks: 61 (limit: 9511)
     Memory: 4.9G
        CPU: 3min 17.469s
     CGroup: /system.slice/elasticsearch.service
             ├─ 461 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.type=deb -cp "/usr/share/elasticsearch/lib/*:/usr/share/>
             ├─1478 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=tr>
             └─1498 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Warning: some journal files were not opened due to insufficient permissions.

Basically only tasks and CPU change between the 3.

But using HTTPs, what should I do something so that these three can communicate?
Disable HTTPs or some other procedure?

4

From what you shared your elasticsearch is running without any issues, it is been up for 44 minutes.

It doesn't look that there is no issue in your cluster.

The nodes doesn't use HTTPs to communicate between them, they use the transport protocol, but the clients need to use https.

You need to curl your cluster using https and authenticating.

curl -XGET 'https://192.168.0.1:9200/_cluster/health?pretty' -k -u elastic

It will them ask for the elastic user password, you need to use the password that was created during installation.

If you didn't remember the password you will need to reset it following the steps in the documentation.

5

Thanks @leandrojmp ,

He returned the following.

I thought that when performing a curl with HTTPs I should authenticate the user by also passing an authentication certificate.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "status_exception",
        "reason" : "Cluster state has not been recovered yet, cannot write to the [null] index"
      }
    ],
    "type" : "authentication_processing_error",
    "reason" : "failed to promote the auto-configured elastic password hash",
    "caused_by" : {
      "type" : "status_exception",
      "reason" : "Cluster state has not been recovered yet, cannot write to the [null] index"
    }
  },
  "status" : 503
}
6

You can pass a certificate with --cacert or use -k to ignore the certificate.

It is not clear what request you made to generate this response, please share the request.

Also, look in Elasticsearch logs for all your nodes and see if there is any WARN or ERROR log lines.

7

@leandrojmp, the request was:

curl -XGET 'https://192.168.0.1:9200/_cluster/health?pretty' -k -u elastic

Elasticsearch log showed these WARN

[2023-06-27T16:25:43,560][WARN ][o.e.d.PeerFinder         ] [master-01] address [192.168.0.3:9300], node [null], requesting [false] discovery result: [][192.168.0.3:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with
 any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors
[2023-06-27T16:25:43,560][WARN ][o.e.d.PeerFinder         ] [master-01] address [192.168.0.2:9300], node [null], requesting [false] discovery result: [][192.168.0.2:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with
 any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors
[2023-06-27T16:25:43,560][WARN ][o.e.t.TcpTransport       ] [master-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/192.168.0.1:58902, remoteAddress=/192.168.0.2:9300, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

[2023-06-27T16:25:42,560][WARN ][o.e.d.PeerFinder         ] [master-01] address [192.168.0.3:9300], node [null], requesting [false] discovery result: [][192.168.0.3:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors
[2023-06-27T16:25:42,561][WARN ][o.e.t.TcpTransport       ] [master-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/192.168.0.1:49296, remoteAddress=/192.168.0.3:9300, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
8

You have some SSL error on the transport certificate, this is an issue as the nodes will note be able to communicate to form the cluster.

[2023-06-27T16:25:42,561][WARN ][o.e.t.TcpTransport ] [master-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/192.168.0.1:49296, remoteAddress=/192.168.0.3:9300, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

How did you create the certificate?

9

I didn't create the certificate.

Should I create it the way you indicate on your blog (I'm also from Brazil)?

10

Vou trocar pro português então.

Esse post é um tutorial pra criar um cluster com 3 nós com a segurança configurada, parece ser o que você quer.

Segue os passos pra criar uma CA e os certificados que deve funcionar.

11

Olá @leandrojmp,

Obrigado, funcionou perfeitamente :smiley:

Thank you, it worked perfectly :smiley:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.