Elasticsearch Won't Start After Upgrading from 5.5 to 6.0

Elastic Stack Elasticsearch
1

We have been unable to get Elasticsearch to start back up after upgrading following this guide: https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html

Elasticsearchhas this status :
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2017-11-17 10:41:30 CST; 1min 0s ago
Docs: http://www.elastic.co
Process: 2739 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2739 (code=exited, status=1/FAILURE)

Nov 17 10:41:30 HQPESDN01 systemd[1]: Started Elasticsearch.
Nov 17 10:41:30 HQPESDN01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 17 10:41:30 HQPESDN01 systemd[1]: elasticsearch.service: Unit entered failed state.
Nov 17 10:41:30 HQPESDN01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

And these errors in the syslog:
Nov 17 10:19:09 HQPESDN01 systemd[1]: Started Elasticsearch.
Nov 17 10:19:09 HQPESDN01 elasticsearch[2488]: ES_PATH_CONF must be set to the configuration path
Nov 17 10:19:09 HQPESDN01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 17 10:19:09 HQPESDN01 systemd[1]: elasticsearch.service: Unit entered failed state.
Nov 17 10:19:09 HQPESDN01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

We set the ES_PATH_CONF as an environment variable to the directory that contains the elasticsearch.yml files and jvm.options files. This is the same location they were located at before the upgrade.

Any help would be amazing as we are struggling to find anything on the error we are getting.
Thanks.

2

There are no errors within Elasticsearch logs as it is not starting up.

3

We tried to see if this was the issue as this user had a similar error message with the Elasticsearch's status:

However we are still seeing the same errors as above.

4

I'm experiencing simlar problems. Unable to start the elasticsearch service. elasticsearch-plugin failes too:

sudo bin/elasticsearch-plugin list
ES_PATH_CONF must be set to the configuration path

5

What OS?
How did you install Elasticsearch, repo or DEB/RPM?

6

Can you please check the contents of /usr/lib/systemd/system/elasticsearch.service and /etc/default/elasticsearch or /etc/sysconfig/elasticsearch (depending on whether or not you installed from the Debian or RPM packages respectively) and verify that ES_PATH_CONF is indeed set and is set to the value where you have your configuration files located (the default is /etc/elasticsearch)?

7

Hello,

For me I'm using CentOS 7 and I update with yum rpm. Before the update I changed this in /etc/sysconfig/elasticsearch:

# Elasticsearch configuration directory
#CONF_DIR=/etc/elasticsearch
ES_PATH_CONF=/etc/elasticsearch

Tried multiple configurations in /usr/lib/systemd/system/elasticsearch.service => This is the relvant part now:

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec

ExecStart=/usr/share/elasticsearch/bin/elasticsearch \
                                                -p ${PID_DIR}/elasticsearch.pid \
                                                --quiet \
                                                -Edefault.path.logs=${LOG_DIR} \
                                                -Edefault.path.data=${DATA_DIR} \
                                                -Edefault.path.conf=${ES_PATH_CONF}

Also tried to omit

                                                -Edefault.path.logs=${LOG_DIR} \
                                                -Edefault.path.data=${DATA_DIR} \
                                                -Edefault.path.conf=${ES_PATH_CONF}

But I still can't start the elasticsearch service. There seems to be no info in /var/log messages.

8

Ok, as a test I installed Elasticsearch 6 on an empty server and compared the /usr/lib/systemd/system/elasticsearch.service and /etc/sysconfig/elasticsearch configuration files.

So on the server where I'm testing the upgrade /etc/sysconfig/elasticsearch has now:

################################
# Elasticsearch
################################

# Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch

# Elasticsearch Java path
#JAVA_HOME=

# Elasticsearch configuration directory
ES_PATH_CONF=/etc/elasticsearch

# Elasticsearch logs directory
#LOG_DIR=/var/log/elasticsearch

# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS
#ES_JAVA_OPTS=

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
# Elasticsearch service
################################

# SysV init.d
#
# When executing the init script, this user will be used to run the elasticsearch service.
# The default value is 'elasticsearch' and is declared in the init.d file.
# Note that this setting is only used by the init script. If changed, make sure that
# the configured user can read and write into the data, work, plugins and log directories.
# For systemd service, the user is usually configured in file /usr/lib/systemd/system/elasticsearch.service
#ES_USER=elasticsearch
#ES_GROUP=elasticsearch

# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5

################################
# System properties
################################

# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65536

# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using Systemd, the LimitMEMLOCK property must be set
# in /usr/lib/systemd/system/elasticsearch.service
MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144

And /usr/lib/systemd/system/elasticsearch.service:

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=elasticsearch
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of bytes of memory that may be locked into RAM
# Set to "infinity" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml and 'MAX_LOCKED_MEMORY=unlimited' in /etc/sysconfig/elasticsearch
LimitMEMLOCK=infinity

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

# Built for distribution-6.0.0 (distribution)

After reloading the daemon I still can't start the elasticsearch service however.

sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2017-11-19 17:03:30 CET; 1min 41s ago
     Docs: http://www.elastic.co
  Process: 6402 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 6402 (code=exited, status=1/FAILURE)

Nov 19 17:03:29 srvelk01 systemd[1]: Started Elasticsearch.
Nov 19 17:03:29 srvelk01 systemd[1]: Starting Elasticsearch...
Nov 19 17:03:30 srvelk01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 19 17:03:30 srvelk01 systemd[1]: Unit elasticsearch.service entered failed state.
Nov 19 17:03:30 srvelk01 systemd[1]: elasticsearch.service failed.

And my elasticsearch.yml:

# ----------------------------------- Cluster ----------------------------------
cluster.name: elastic_dv01
# ----------------------------------- Node -------------------------------------
node.name: ${HOSTNAME}
node.master: true
node.data: true
node.ingest: true
node.attr.datacenter: AC
# ----------------------------------- Paths ------------------------------------
path.data: /datadisk/elasticsearch
path.logs: /logdisk/elasticsearch
path.repo: /mnt/snapshots/elastic_dv01
# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
# ----------------------------------- Network ----------------------------------
network.host: [_local_]
http.port: 9200

Grtz

Willem

9

Some extra info. For some reason /var/run/elasticsearch no longer exists once elasticsearch is upgraded with yum rpm. Eve weirder is that when I attempt to create it manually and try to start the service, somehow the folder is deleted...

So:

sudo mkdir /var/run/elasticsearch
sudo chmod 755 /var/run/elasticsearch
sudo systemctl start elasticsearch

Results:

sudo ls -la /var/run/elasticsearch
ls: cannot access /var/run/elasticsearch: No such file or directory

Grtz

10

Good news, I found the remaining problem. Still had to remove and reinstall x-pack after making the changes in /usr/lib/systemd/system/elasticsearch.service and /etc/sysconfig/elasticsearch

So shouldn't an rpm yum update not automatically update /usr/lib/systemd/system/elasticsearch.service and /etc/sysconfig/elasticsearch? Or are we supposed update these files ourselves?

1 Like
11

We are running it on Ubuntu 16.04.2 LTS. We installed it using the DEB package. We set ES_PATH_CONF as environment variable where the value is set to /etc/elasticsearch. Which is where we have the configuration files. From what willemdh said I believe that updating elasticsearch 5.5.0 to 6.0.0 using the DEB package is not updating the /usr/lib/systmd/system/elasticsearch.service. I compared the files from the previous 5.5.0 files and they have not changed. Which is causing the issue because variable names have changed according to this resolved issue in 6.0.0: https://github.com/elastic/elasticsearch/pull/26197

Thus I believe that the installer even when run from root doesn't fully change files correctly. Is this a bug or are we supposed to change these files ourselves? If we are supposed to change them ourselves, can some one give me a default configuration for 6.0.0 for those files so that I can change them?

12

We have tried to update the node again from 5.5 but this time we noticed we need to say yes to the changeing of the /usr/lib/systemd/system/elasticsearch.service file. Thus we did it. The files /usr/lib/systemd/system/elasticsearch.service and /etc/default/elasticsearch are both set correctly. We didn't change the default location for the configuration files. The service is still failing with this error message:

● elasticsearch.service - Elasticsearch
  Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2017-11-21 09:57:21 CST; 2s ago
  Docs: http://www.elastic.co
Process: 3937 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet 
(code=exited, status=1/FAILURE)
Main PID: 3937 (code=exited, status=1/FAILURE)

Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,184 main ERROR Null object returned for 
RollingFile in Appenders.
Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,184 main ERROR Null object returned for 
RollingFile in Appenders.
Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,184 main ERROR Unable to locate 
appender "rolling" for logger config "root"
Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,185 main ERROR Unable to locate 
appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,185 main ERROR Unable to locate 
appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile
Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,185 main ERROR Unable to locate 
appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Nov 21 09:57:21 HQPESCN02 elasticsearch[3937]: 2017-11-21 09:57:21,185 main ERROR Unable to locate 
appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Nov 21 09:57:21 HQPESCN02 systemd[1]: elasticsearch.service: Main process exited, code=exited, 
status=1/FAILURE
Nov 21 09:57:21 HQPESCN02 systemd[1]: elasticsearch.service: Unit entered failed state.
Nov 21 09:57:21 HQPESCN02 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

It starts to run then it crashes. Any help with this would be great.

14

We got a clone ingest node to finally start up. I noticed that it finally was posting logs to the /var/log/elasticsearch. Within those logs there was an exception occurring where the address would not bind.
Inside of our /etc/elasticsearch/elasticsearch.yml, we had the network.host: [the ip for this machine] and when we removed that from the file it started up. So to recap the process we went through.

  1. Updated the node using debian package. Say no to the overwriting of the config files except for the elasticsearch.service file this one needs to change because global variables have changed (if you have edited this file in the past then go into the directory and the old one will still be there just change the values of the variables to whatever you set it as with the corresponding new variable).
  2. removed the x-pack plugin and then reinstalled it saying yes to everything it needs.
  3. Went into the elasticsearch.yml and removed the network.host: [ip] I believe this is only need to be done when you are getting an error within your elasticsearch logs about being unable to bind to that specific ip address.

I'm not sure what changed between 5.5 and 6.0, but the network.host is needed for 5.5 and not needed for 6.0.

This was only a ingest node and thus had no data on it. Next week we are going to attempt the updating on the data nodes. Otherwise this was a fix for our ingest nodes. Any questions about this just ask Thanks!

15

It is needed if you want to listen on interfaces other than localhost.

1 Like
16

That is interesting. We are still able to listen on interfaces other than the localhost.... Its weird we tried removing it in 5.5 and ran a netstat -ntulp which came back with the normal response of it only being on the localhost. But after doing the update process I stated above. Elasticsearch was unable to start and the logs said it was failing to bind to that address. So we removed that network.host from the yml and Elasticsearch started up. We ran a netstat -ntulp and it said elasticsearch was on the [IP]:9200 interface. We have http enabled on that node so we checked from a browser on our computer (not where elasticsearch is being run from) and sure enough the node was able to be connected to from that [IP] on port 9200. Hm.... I don't understand why this is occurring.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.