# ElasticSearch does not start and not even cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 27, 2023, 6:24pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017 "2023-06-27T18:24:55Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![TomTom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomtom/32/3676_2.png) [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Post date:** [June 27, 2023, 6:24pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/1 "2023-06-27T18:24:56Z")
I'm trying to create an Elasticsearch cluster with 3 nodes, each node being eligible as a master, as stated in this [doc](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/high-availability-cluster-small-clusters.html). This cluster will be used by the end user only on our local network through an nginx proxy that will point his port 9200 to port 9200 of one of the three nodes. For that, I configured each node to use `network.host` with the fixed local IP of each machine. It turns out that even locally when running the command `curl -XGET '192.168.0.1:9200/_cluster/health?pretty'` I just get the response > curl: (52) Empty reply from server. What could I have done wrong? These are the steps I did for configuration and how my configuration files turned out: 1 - I installed ES using Debian packages and kept certificates as it generated, and user as well. 2 - After installing, run the commands below: ```auto sudo mkdir -p /etc/systemd/system/elasticsearch.service.d/ sudo nano /etc/systemd/system/elasticsearch.service.d/override.conf ``` ...and added the line... ```auto [Service] LimitMEMLOCK=infinity ``` 3 - I modified jvm.options to use 4GB of RAM, as the server has 8GB of RAM `sudo nano /etc/elasticsearch/jvm.options` ```auto -Xms4g -Xmx4g ``` 4 - I edited the `elasticsearch.yml` file with the following settings: ```auto cluster.name: my-elk-cluster node.name: master-01 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.0.1 discovery.seed_hosts: ["192.168.0.1", "192.168.0.2", "192.168.0.3"] cluster.initial_master_nodes: ["master-01", "master-02", "master-03"] # Below automatically generated by Elastic on install xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 ``` 5 - And after that, I just configured Elasticsearch to start with the server: ```auto sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service sudo systemctl start elasticsearch.service ``` The command `sudo journalctl --unit elasticsearch` does not show any errors. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 27, 2023, 6:31pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/2 "2023-06-27T18:31:51Z")
What does `systemctl status elasticsearch` return on your nodes? Also, you have security enabled, you need to both use `https` and authenticate with the password created during the installation. ---
**Author:** ![TomTom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomtom/32/3676_2.png) [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Post date:** [June 27, 2023, 6:45pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/3 "2023-06-27T18:45:06Z")
Hi @leandrojmp, This is the return ```auto ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/elasticsearch.service.d └─override.conf Active: active (running) since Tue 2023-06-27 14:57:11 -03; 44min ago Docs: https://www.elastic.co Main PID: 461 (java) Tasks: 61 (limit: 9511) Memory: 4.9G CPU: 3min 17.469s CGroup: /system.slice/elasticsearch.service ├─ 461 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.type=deb -cp "/usr/share/elasticsearch/lib/*:/usr/share/> ├─1478 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=tr> └─1498 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller Warning: some journal files were not opened due to insufficient permissions. ``` Basically only tasks and CPU change between the 3. But using HTTPs, what should I do something so that these three can communicate? Disable HTTPs or some other procedure? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 27, 2023, 7:00pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/4 "2023-06-27T19:00:12Z")
From what you shared your elasticsearch is running without any issues, it is been up for 44 minutes. It doesn't look that there is no issue in your cluster. > [@TomTom](#): > > But using HTTPs, what should I do something so that these three can communicate? The nodes doesn't use HTTPs to communicate between them, they use the transport protocol, but the clients need to use https. You need to curl your cluster using `https` and authenticating. ```auto curl -XGET 'https://192.168.0.1:9200/_cluster/health?pretty' -k -u elastic ``` It will them ask for the `elastic` user password, you need to use the password that was created during installation. If you didn't remember the password you will need to reset it following the steps in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/reset-password.html#_examples_8). ---
**Author:** ![TomTom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomtom/32/3676_2.png) [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Post date:** [June 27, 2023, 7:15pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/5 "2023-06-27T19:15:05Z")
Thanks @leandrojmp , He returned the following. I thought that when performing a curl with HTTPs I should authenticate the user by also passing an authentication certificate. ```auto { "error" : { "root_cause" : [ { "type" : "status_exception", "reason" : "Cluster state has not been recovered yet, cannot write to the [null] index" } ], "type" : "authentication_processing_error", "reason" : "failed to promote the auto-configured elastic password hash", "caused_by" : { "type" : "status_exception", "reason" : "Cluster state has not been recovered yet, cannot write to the [null] index" } }, "status" : 503 } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 27, 2023, 7:21pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/6 "2023-06-27T19:21:36Z")
> [@TomTom](#): > > I thought that when performing a curl with HTTPs I should authenticate the user by also passing an authentication certificate. You can pass a certificate with `--cacert` or use `-k` to ignore the certificate. It is not clear what request you made to generate this response, please share the request. Also, look in Elasticsearch logs for all your nodes and see if there is any WARN or ERROR log lines. ---
**Author:** ![TomTom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomtom/32/3676_2.png) [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Post date:** [June 27, 2023, 7:28pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/7 "2023-06-27T19:28:33Z")
@leandrojmp, the request was: `curl -XGET 'https://192.168.0.1:9200/_cluster/health?pretty' -k -u elastic` Elasticsearch log showed these WARN ```auto [2023-06-27T16:25:43,560][WARN][o.e.d.PeerFinder] [master-01] address [192.168.0.3:9300], node [null], requesting [false] discovery result: [][192.168.0.3:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors [2023-06-27T16:25:43,560][WARN][o.e.d.PeerFinder] [master-01] address [192.168.0.2:9300], node [null], requesting [false] discovery result: [][192.168.0.2:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors [2023-06-27T16:25:43,560][WARN][o.e.t.TcpTransport] [master-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/192.168.0.1:58902, remoteAddress=/192.168.0.2:9300, profile=default}], closing connection io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors [2023-06-27T16:25:42,560][WARN][o.e.d.PeerFinder] [master-01] address [192.168.0.3:9300], node [null], requesting [false] discovery result: [][192.168.0.3:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors [2023-06-27T16:25:42,561][WARN][o.e.t.TcpTransport] [master-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/192.168.0.1:49296, remoteAddress=/192.168.0.3:9300, profile=default}], closing connection io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 27, 2023, 8:27pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/8 "2023-06-27T20:27:39Z")
You have some SSL error on the transport certificate, this is an issue as the nodes will note be able to communicate to form the cluster. > [2023-06-27T16:25:42,561][WARN][o.e.t.TcpTransport] [master-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/192.168.0.1:49296, remoteAddress=/192.168.0.3:9300, profile=default}], closing connection > io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors How did you create the certificate? ---
**Author:** ![TomTom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomtom/32/3676_2.png) [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Post date:** [June 27, 2023, 8:45pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/9 "2023-06-27T20:45:45Z")
I didn't create the certificate. Should I create it the way you indicate on your [blog](https://web.leandrojmp.com/posts/pt-br/2022/10/elasticsearch-security) (I'm also from Brazil)? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 27, 2023, 9:36pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/10 "2023-06-27T21:36:06Z")
> [@TomTom](#): > > Should I create it the way you indicate on your [blog](https://web.leandrojmp.com/posts/pt-br/2022/10/elasticsearch-security) (I'm also from Brazil)? Vou trocar pro português então. Esse post é um tutorial pra criar um cluster com 3 nós com a segurança configurada, parece ser o que você quer. Segue os passos pra criar uma CA e os certificados que deve funcionar. ---
**Author:** ![TomTom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomtom/32/3676_2.png) [@TomTom](https://discuss.elastic.co/u/TomTom)\ **Post date:** [June 29, 2023, 6:29pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/11 "2023-06-29T18:29:46Z")
Olá @leandrojmp, Obrigado, funcionou perfeitamente 😃 _Thank you, it worked perfectly 😃_ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 27, 2023, 6:30pm UTC](https://discuss.elastic.co/t/elasticsearch-does-not-start-and-not-even-cluster/337017/12 "2023-07-27T18:30:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.