Elasticsearch node does not starts

[pszemesy (Peter Szemesy)]

Hi All,

I have a problem with starting one of my nodes.
The ES runs on a cluster of 3 nodes (version 6.6).
The node2 dosen't want to start up.
When I start it from systemctl it seems to be running:

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-02-14 14:41:15 CET; 4s ago
Docs: http://www.elastic.co
Main PID: 20149 (java)
CGroup: /system.slice/elasticsearch.service
└─20149 /bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless...

Feb 14 14:41:15 aspnaplod2 systemd[1]: Started Elasticsearch.

But in few seconds it fails - without writing any character of logs.

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-02-14 14:41:21 CET; 489ms ago
Docs: http://www.elastic.co
Process: 20149 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 20149 (code=exited, status=1/FAILURE)

Feb 14 14:41:15 aspnaplod2 systemd[1]: Started Elasticsearch.
Feb 14 14:41:21 aspnaplod2 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Feb 14 14:41:21 aspnaplod2 systemd[1]: Unit elasticsearch.service entered failed state.
Feb 14 14:41:21 aspnaplod2 systemd[1]: elasticsearch.service failed.

Have you got any idea why it could happened?

[Ant]

is this a sepcific node or any time you try to start a second node?

If any time you try and start a second node if you stop the first node are one of the other nodes able to start?

what happens if you query the clusters health?

curl -XGET -u {elasticuser}:{elasticpassword} "{elastichost}:9200/_cluster/health"

Is this a newely built cluster?

Have you made any changes recently?

Have you tried to explicity set a log file in the elasticsearch.yml file to see if anything populates?

[pszemesy (Peter Szemesy)]

Hi,
Yes, the other nodes starts correctly. Health state is yellow now (unable to allocate some replica shards).
This cluster is implemented in last November. The new addition in the elasticsearch.yml is the auditing extra added (xpack.security.audit.logfile.events.emit_request_body: true), although this addition is done on the other nodes also. On the node-2 there are some other components running without any problem (filebeat, metricbeat, logstash).
The logfile location is set in the elasticsearch.yml (to a folder on different disk) - it is done at the beginning.

[Ant]

Certainly sounds a node specific issue then rather than the cluster as a whole. If you have made changes to the elasticsearch.yml file I'd start by rolling back to the old version that worked if you still have it and re-applying the change incase an error was introduced. if not, load up one of the known good ones side by side with your one that isn't working. Obviously there will be some differences for things like hostname, ip etc. but check everything to make sure you don't have a formating issue in there as it could be one entry is set wrong and throwing out everything else. Alternatly you can take the file from a good node and change the node specific variables for the one that is faulting. Keep a backup of the old file for reference if you do this.

It's been a while since I had to do it so I can't remember the process but you could look up how to lauch it in a session rather than as a service so you see the errors on screen.

It's drastic but if you really can't find your needle in a haystack as to why this isn't working there is always the option to rebuild the node, re-introduce it to the cluster and let it all ballance out.

[Tek_Chand]

@pszemesy, Can you please share your elasticsearch.yml config for the machine on which you are facing issue?

Thanks.

[pszemesy (Peter Szemesy)]

Please find it below:

cluster.name: asp_development
node.name: aspnaplod2
path.data: /data/elasticsearch
path.logs: /data/log/elasticsearch
network.host: _site_
http.port: 9200
discovery.zen.ping.unicast.hosts: ["999.999.999.141", "999.999.999.143"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/cert/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/cert/elastic-certificates.p12
xpack.notification.email.account:
  idom_account:
    profile: standard
    smtp:
      host: qwert.qwert.hu
      port: 25
      user: asdf.asdf@qwert.hu
xpack.security.audit.enabled: true
xpack.security.audit.outputs: [ index, logfile ]
xpack.security.audit.logfile.events.emit_request_body: true