ES Service Will Not Start After 5.0 Upgrade on CentOS

Sophitia · October 27, 2016, 4:08pm

I am new to ES. I have a single node running 2.4 as a POC. I upgraded my 2.4 node to 5.0 but the service will not start. Here is some additional information

service elasticsearch status -v
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2016-10-27 12:20:17 EDT; 1s ago
Docs: http://www.elastic.co
Process: 4784 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 4783 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 4784 (code=exited, status=1/FAILURE)
Oct 27 12:20:17 ELK01 systemd[1]: Starting Elasticsearch...
Oct 27 12:20:17 ELK01 systemd[1]: Started Elasticsearch.
Oct 27 12:20:17 ELK01 elasticsearch[4784]: Error: encountered environment variables that are no longer supported
Oct 27 12:20:17 ELK01 elasticsearch[4784]: Use jvm.options or ES_JAVA_OPTS to configure the JVM
Oct 27 12:20:17 ELK01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 27 12:20:17 ELK01 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 27 12:20:17 ELK01 systemd[1]: elasticsearch.service failed.

journalctl -u elasticsearch.service
Oct 27 12:12:36 ELK01 systemd[1]: Starting Elasticsearch...
Oct 27 12:12:36 ELK01 systemd[1]: Started Elasticsearch.
Oct 27 12:12:36 ELK01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 27 12:12:36 ELK01 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 27 12:12:36 ELK01 systemd[1]: elasticsearch.service failed.
Oct 27 12:16:42 ELK01 systemd[1]: Starting Elasticsearch...
Oct 27 12:16:42 ELK01 systemd[1]: Started Elasticsearch.
Oct 27 12:16:42 ELK01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 27 12:16:42 ELK01 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 27 12:16:42 ELK01 systemd[1]: elasticsearch.service failed.

Any direction here would be greatly appreciated.

xavierfacq · October 27, 2016, 5:49pm

You should search in the Elasticsearch log , probably in /var/log/elasticsearch.

Sophitia · October 27, 2016, 6:01pm

Thanks for your suggestion. However, the log was not very helpful.

I put it down and came back a bit later. I was able to resolve that issue by:

Add options in /etc/elasticsearch/jvm.options for my heap
-xms4g
-xmx4g
Commented out in /etc/sysconfig/elasticsearch
-# ES_HEAP_SIZE=4g

ES service now starts. I seem to have a binding issue now, but I'll save that for a new thread.

I hope this will help additional upgraders!

EDIT: Looks like the service started but immediately stop.

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2016-10-27 14:04:37 EDT; 24s ago
Docs: http://www.elastic.co
Process: 5776 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=64)
Process: 5775 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 5776 (code=exited, status=64)

Oct 27 14:04:36 ELK01 systemd[1]: Starting Elasticsearch...
Oct 27 14:04:36 ELK01 systemd[1]: Started Elasticsearch.
Oct 27 14:04:37 ELK01 elasticsearch[5776]: starts elasticsearch
Oct 27 14:04:37 ELK01 elasticsearch[5776]: Option Description
Oct 27 14:04:37 ELK01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=64/n/a
Oct 27 14:04:37 ELK01 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 27 14:04:37 ELK01 systemd[1]: elasticsearch.service failed.

xavierfacq · October 27, 2016, 6:13pm

do you have log into this directory ?

Sophitia · October 27, 2016, 7:28pm

Yep. This is from /var/log/elasticsearch/elasticsearch.log

[2016-10-27 10:30:00,445][INFO ][cluster.routing.allocation.decider] [ELK01] updating [cluster.routing.allocation.enable] from [ALL] to [NONE]
[2016-10-27 10:35:47,166][INFO ][node ] [ELK01] stopping ...
[2016-10-27 10:35:48,272][INFO ][node ] [ELK01] stopped
[2016-10-27 10:35:48,273][INFO ][node ] [ELK01] closing ...
[2016-10-27 10:35:48,290][INFO ][node ] [ELK01] closed

I had been following the full restart documentation: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/restart-upgrade.html

Sophitia · October 27, 2016, 8:16pm

I am continuing to work on this. I was able to get the service to start by editing /usr/share/elasticsearch/bin/elasticsearch.

I uncommented ES_JAVA_OPTS and updated the default heap to my desired size. Here is the full line:
ES_JAVA_OPTS=-"Xms4g -Xmx4g" ./bin/elasticsearch

Checking the status takes quite some time. There are many PIDs.
Oct 27 16:11:33 ELK01 elasticsearch[7458]: # There is insufficient memory for the Java Runtime Environment to continue.
Oct 27 16:11:33 ELK01 elasticsearch[7458]: # Native memory allocation (mmap) failed to map 4120510464 bytes for committing reserved memory.
Oct 27 16:11:33 ELK01 elasticsearch[7458]: # An error report file with more information is saved as:
Oct 27 16:11:33 ELK01 elasticsearch[7458]: # /tmp/hs_err_pid52983.log
Oct 27 16:11:34 ELK01 elasticsearch[7458]: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006ca660000, 4120510464, 0) failed; error='Cannot allocate memory' (errno=12)
Oct 27 16:11:34 ELK01 elasticsearch[7458]: #
Oct 27 16:11:34 ELK01 elasticsearch[7458]: # There is insufficient memory for the Java Runtime Environment to continue.
Oct 27 16:11:34 ELK01 elasticsearch[7458]: # Native memory allocation (mmap) failed to map 4120510464 bytes for committing reserved memory.
Oct 27 16:11:34 ELK01 elasticsearch[7458]: # An error report file with more information is saved as:
Oct 27 16:11:34 ELK01 elasticsearch[7458]: # /tmp/hs_err_pid52982.log

So it looks like it has turned into a JVM issue. This is a tiny POC to become familiar with the product. Kibana and Logstash are also running on this server. It has 12GB of RAM total. Before the upgrade, I had 4GB dedicated to Logstash heap and 4GB for ElasticSearch heap.

My VM has become unresponsive and I can no longer SSH in. Pending a hard reboot from the admin.

warkolm · October 27, 2016, 11:48pm

Don't do that, use the environment files to handle those settings.