ES master only works and 2 other nodes get empty cluster id

i created 3 ES machines in virtualbox
1 master1
1 data1
1 data2

my master01 only get cluster_uuid

two other nodes failed "cluster_uuid": "_na_""

master

cluster.name: mycluster
node.name: master1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.123.0.1", "192.123.0.2", "192.123.0.2"]
cluster.initial_master_nodes: ["192.123.0.1"]

data1

cluster.name: mycluster
node.name: data1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.123.0.1", "192.123.0.2", "192.123.0.2"]
cluster.initial_master_nodes: ["192.123.0.1"]

data2

cluster.name: mycluster
node.name: data2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.123.0.1", "192.123.0.2", "192.123.0.2"]
cluster.initial_master_nodes: ["192.123.0.1"]

the error shows data1 and data2

{

  "error" : {

    "root_cause" : [

      {

        "type" : "master_not_discovered_exception",

        "reason" : null

      }

    ],

    "type" : "master_not_discovered_exception",

    "reason" : null

  },

  "status" : 503

}

Should I change any port to the same as the build master for this image?
Screenshot 2022-05-28 025411

Welcome to our community! :smiley:

Can the nodes all connect to each other within the virtualbox environment?

hai @warkolm

currently i am getting this error

Exception in thread "main" ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - truststore file [/etc/elasticsearch/certs/elastic-certificates.p12] does not exist]; nested: NoSuchFileException[/etc/elasticsearch/certs/elastic-certificates.p12];

	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:550)

	at java.base/java.util.HashMap.forEach(HashMap.java:1421)

	at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)

	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:546)

	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:159)

	at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.execute(CommandLineHttpClient.java:92)

	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:327)

	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$InteractiveSetup.execute(SetupPasswordTool.java:199)

	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)

	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)

	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)

	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)

	at org.elasticsearch.cli.Command.main(Command.java:77)

	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:128)

Caused by: ElasticsearchException[failed to initialize SSL TrustManager - truststore file [/etc/elasticsearch/certs/elastic-certificates.p12] does not exist]; nested: NoSuchFileException[/etc/elasticsearch/certs/elastic-certificates.p12];

	at org.elasticsearch.xpack.core.ssl.TrustConfig.missingTrustConfigFile(TrustConfig.java:117)

	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:69)

	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:455)

	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1220)

	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:548)

	... 13 more

Caused by: java.nio.file.NoSuchFileException: /etc/elasticsearch/certs/elastic-certificates.p12

	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)

	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)

	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)

	at java.base/java.nio.file.Files.newByteChannel(Files.java:380)

	at java.base/java.nio.file.Files.newByteChannel(Files.java:432)

	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)

	at java.base/java.nio.file.Files.newInputStream(Files.java:160)

	at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:97)

	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66)

Does it exist?

@warkolm
i completely removed the install again but i get this error

fatal: [es1]: FAILED! => {"changed": true, "cmd": "/usr/share/elasticsearch/bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass \"\"", "delta": "0:00:04.039244", "end": "2022-06-04 07:43:14.962140", "msg": "non-zero return code", "rc": 74, "start": "2022-06-04 07:43:10.922896", "stderr": "\nERROR: Output file '/tmp/elastic-certificates.p12' already exists", "stderr_lines": ["", "ERROR: Output file '/tmp/elastic-certificates.p12' already exists"], "stdout": "This tool assists you in the generation of X.509 certificates and certificate\nsigning requests for use with SSL/TLS in the Elastic stack.\n\nThe 'cert' mode generates X.509 certificate and private keys.\n    * By default, this generates a single certificate and key for use\n       on a single instance.\n    * The '-multiple' option will prompt you to enter details for multiple\n       instances and will generate a certificate and key for each one\n    * The '-in' option allows for the certificate generation to be automated by describing\n       the details of each instance in a YAML file\n\n    * An instance is any piece of the Elastic Stack that requires an SSL certificate.\n      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats\n      may all require a certificate and private key.\n    * The minimum required value for each instance is a name. This can simply be the\n      hostname, which will be used as the Common Name of the certificate. A full\n      distinguished name may also be used.\n    * A filename value may be required for each instance. This is necessary when the\n      name would result in an invalid file or directory name. The name provided here\n      is used as the directory name (within the zip) and the prefix for the key and\n      certificate files. The filename is required if you are prompted and the name\n      is not displayed in the prompt.\n    * IP addresses and DNS names are optional. Multiple values can be specified as a\n      comma separated string. If no IP addresses or DNS names are provided, you may\n      disable hostname verification in your SSL configuration.\n\n    * All certificates generated by this tool will be signed by a certificate authority (CA)\n      unless the --self-signed command line option is specified.\n      The tool can automatically generate a new CA for you, or you can provide your own with\n      the --ca or --ca-cert command line options.\n\nBy default the 'cert' mode produces a single PKCS#12 output file which holds:\n    * The instance certificate\n    * The private key for the instance certificate\n    * The CA certificate\n\nIf you specify any of the following options:\n    * -pem (PEM formatted output)\n    * -keep-ca-key (retain generated CA key)\n    * -multiple (generate multiple certificates)\n    * -in (generate certificates from an input file)\nthen the output will be be a zip file containing individual certificate/key files\n\nNote: Generating certificates without providing a CA certificate is deprecated.\n      A CA certificate will become mandatory in the next major release.", "stdout_lines": ["This tool assists you in the generation of X.509 certificates and certificate", "signing requests for use with SSL/TLS in the Elastic stack.", "", "The 'cert' mode generates X.509 certificate and private keys.", "    * By default, this generates a single certificate and key for use", "       on a single instance.", "    * The '-multiple' option will prompt you to enter details for multiple", "       instances and will generate a certificate and key for each one", "    * The '-in' option allows for the certificate generation to be automated by describing", "       the details of each instance in a YAML file", "", "    * An instance is any piece of the Elastic Stack that requires an SSL certificate.", "      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats", "      may all require a certificate and private key.", "    * The minimum required value for each instance is a name. This can simply be the", "      hostname, which will be used as the Common Name of the certificate. A full", "      distinguished name may also be used.", "    * A filename value may be required for each instance. This is necessary when the", "      name would result in an invalid file or directory name. The name provided here", "      is used as the directory name (within the zip) and the prefix for the key and", "      certificate files. The filename is required if you are prompted and the name", "      is not displayed in the prompt.", "    * IP addresses and DNS names are optional. Multiple values can be specified as a", "      comma separated string. If no IP addresses or DNS names are provided, you may", "      disable hostname verification in your SSL configuration.", "", "    * All certificates generated by this tool will be signed by a certificate authority (CA)", "      unless the --self-signed command line option is specified.", "      The tool can automatically generate a new CA for you, or you can provide your own with", "      the --ca or --ca-cert command line options.", "", "By default the 'cert' mode produces a single PKCS#12 output file which holds:", "    * The instance certificate", "    * The private key for the instance certificate", "    * The CA certificate", "", "If you specify any of the following options:", "    * -pem (PEM formatted output)", "    * -keep-ca-key (retain generated CA key)", "    * -multiple (generate multiple certificates)", "    * -in (generate certificates from an input file)", "then the output will be be a zip file containing individual certificate/key files", "", "Note: Generating certificates without providing a CA certificate is deprecated.", "      A CA certificate will become mandatory in the next major release."]}


I used an ansible script to run this Elasticsearch because i changed jvm.options using ram xs512m so i run this sctipt again which means it shows this error so its already created so how can i remove it and create it again ?

my log error:

-known cluster state; node term 0, last-accepted version 0 in term 0

[2022-06-04T07:49:16,949][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}]; discovery will continue using [192.168.0.210:9300, 192.168.0.142:9300] from hosts providers and [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

[2022-06-04T07:49:26,951][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}]; discovery will continue using [192.168.0.210:9300, 192.168.0.142:9300] from hosts providers and [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

[2022-06-04T07:49:29,455][WARN ][o.e.d.PeerFinder         ] [node-1] address [192.168.0.142:9300], node [null], requesting [false] connection failed: [][192.168.0.142:9300] connect_timeout[30s]

[2022-06-04T07:49:29,456][WARN ][o.e.d.PeerFinder         ] [node-1] address [192.168.0.210:9300], node [null], requesting [false] connection failed: [][192.168.0.210:9300] connect_timeout[30s]

[2022-06-04T07:49:36,953][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}]; discovery will continue using [192.168.0.210:9300, 192.168.0.142:9300] from hosts providers and [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

[2022-06-04T07:49:46,956][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}]; discovery will continue using [192.168.0.210:9300, 192.168.0.142:9300] from hosts providers and [{node-1}{7sCqfbtxRaCAabxjea_ItQ}{Agc-bKq8SLO90Md5rG9nBw}{192.168.0.245}{192.168.0.245:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.