Elasticsearch service doesn't start upon system reboot

Hi,
My environement is below
Elastic Version - "2.3.5"
OS - RHEL 7
JAVA Version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)

Command to start ES - Systemctl start elasticsearch

Issue -
Whenever I reboot my host server, elastic search fails to start with below error:

Aug 14 06:43:00 ********systemd: Starting Elasticsearch...
Aug 14 06:43:00 ******** systemd: Started Elasticsearch.
Aug 14 06:43:01 ******** elasticsearch: Exception in thread "main" java.nio.file.AccessDeniedException: /var/run/elasticsearch
Aug 14 06:43:01 systemd: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Aug 14 06:43:01
elasticsearch: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Aug 14 06:43:01 ******** elasticsearch: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Aug 14 06:43:01 ******** elasticsearch: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Aug 14 06:43:01 ******** elasticsearch: at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
Aug 14 06:43:01 elasticsearch: at java.nio.file.Files.createDirectory(Files.java:674)
Aug 14 06:43:01
elasticsearch: at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
Aug 14 06:43:01 ********elasticsearch: at java.nio.file.Files.createDirectories(Files.java:767)
Aug 14 06:43:01 ******** elasticsearch: at org.elasticsearch.common.PidFile.create(PidFile.java:69)
Aug 14 06:43:01 ********elasticsearch: at org.elasticsearch.common.PidFile.create(PidFile.java:55)
Aug 14 06:43:01 ********elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:247)
Aug 14 06:43:01 ******** elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Aug 14 06:43:01 ********elasticsearch: Refer to the log for complete error details.
Aug 14 06:43:01 ******** systemd: Unit elasticsearch.service entered failed state.
Aug 14 06:43:01 ******** systemd: elasticsearch.service failed.

But if i create /var/run/elasticsearch and chown to elasticsearch user. Elasticsearch service starts normally a nd works fine.

Below is my init script:
#!/bin/sh

elasticsearch

chkconfig: 2345 80 20

description: Starts and stops a single elasticsearch instance on this system

BEGIN INIT INFO

Provides: Elasticsearch

Required-Start: $network $named

Required-Stop: $network $named

Default-Start: 2 3 4 5

Default-Stop: 0 1 6

Short-Description: This service manages the elasticsearch daemon

Description: Elasticsearch is a very scalable, schema-free and high-performance search solution supporting multi-tenancy and near realtime search.

END INIT INFO

init.d / servicectl compatibility (openSUSE)

if [ -f /etc/rc.status ]; then
. /etc/rc.status
rc_reset
fi

Source function library.

if [ -f /etc/rc.d/init.d/functions ]; then
. /etc/rc.d/init.d/functions
fi

Sets the default values for elasticsearch variables used in this script

ES_USER=“elasticsearch”
ES_GROUP=“elasticsearch”
ES_HOME=“/usr/share/elasticsearch”
MAX_OPEN_FILES=65535
MAX_MAP_COUNT=262144
LOG_DIR=“/amex/logs/elasticsearch”
DATA_DIR=“/amex/elasticsearch/data”
CONF_DIR=“/etc/elasticsearch”
PID_DIR=“/var/run/elasticsearch”

Source the default env file

ES_ENV_FILE=“/etc/sysconfig/elasticsearch”
if [ -f “$ES_ENV_FILE” ]; then
. “$ES_ENV_FILE”
fi

CONF_FILE setting was removed

if [ ! -z “$CONF_FILE” ]; then
echo “CONF_FILE setting is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed.”
exit 1
fi
exec=“$ES_HOME/bin/elasticsearch”
prog=“elasticsearch”
pidfile=“$PID_DIR/${prog}.pid”
export ES_HEAP_SIZE
export ES_HEAP_NEWSIZE
export ES_DIRECT_SIZE
export ES_JAVA_OPTS
export ES_GC_LOG_FILE
export ES_STARTUP_SLEEP_TIME
export JAVA_HOME
export ES_INCLUDE
lockfile=/var/lock/subsys/$prog

backwards compatibility for old config sysconfig files, pre 0.90.1

if [ -n $USER ] && [ -z $ES_USER ] ; then
ES_USER=$USER
fi
checkJava() {
if [ -x “$JAVA_HOME/bin/java” ]; then
JAVA=“$JAVA_HOME/bin/java”
else
JAVA=which java
fi
if [ ! -x “$JAVA” ]; then
echo “Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME”
exit 1
fi
}
start() {
checkJava
[ -x $exec ] || exit 5
if [ -n “$MAX_LOCKED_MEMORY” -a -z “$ES_HEAP_SIZE” ]; then
echo “MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set”
return 7
fi
if [ -n “$MAX_OPEN_FILES” ]; then
ulimit -n $MAX_OPEN_FILES
fi
if [ -n “$MAX_LOCKED_MEMORY” ]; then
ulimit -l $MAX_LOCKED_MEMORY
fi
if [ -n “$MAX_MAP_COUNT” -a -f /proc/sys/vm/max_map_count ]; then
sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT
fi
export ES_GC_LOG_FILE
# Ensure that the PID_DIR exists (it is cleaned at OS startup time)
if [ -n “$PID_DIR” ] && [ ! -e “$PID_DIR” ]; then
mkdir -p “$PID_DIR” && chown “$ES_USER”:“$ES_GROUP” “$PID_DIR”
fi
if [ -n “$pidfile” ] && [ ! -e “$pidfile” ]; then
touch “$pidfile” && chown “$ES_USER”:“$ES_GROUP” “$pidfile”
fi
cd $ES_HOME
echo -n $“Starting $prog: ”
# if not running, start it up here, usually something like “daemon $exec”
daemon --user $ES_USER --pidfile $pidfile $exec -p $pidfile -d -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.conf=$CONF_DIR
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $“Stopping $prog: ”
# stop it here, often “killproc $prog”
killproc -p $pidfile -d 86400 $prog
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
stop
start
}
reload() {
restart
}
force_reload() {
restart
}
rh_status() {
# run checks to determine if the service is running or use generic status
status -p $pidfile $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case “$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
restart
;;
*)
echo $“Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}”
exit 2
esac
exit $?

Please let me know if I am doing something wrong.

It looks like your systemd-tmpfiles-setup service is not running on startup which would create the directory at startup with the correct permissions because we provide /usr/lib/tmpfiles.d/elasticsearch.conf that uses this service to do this. I think that you need to verify and fix this.

Thanks for reply Jason.
Yes systemd-tmpfiles-setup was failing because it was not able to find "elasticsearch" user. After further troubleshooting I found that in our environement "elasticsearch" user has been created by someone in LDAP and during boot looks like systemd-tmpfiles-setup is kicking off before power broker service.

Now I am planning to use some different user to run elasticsearch service. It would be great if you point me to any document which talks about using different than default elastucsearch to run elasticsearch service.

Thanks once again!

I'm sorry to tell you that since we do not support that (running as a user other than the elasticsearch user) I'm not aware of any such documentation. I'm glad that we've resolved the issue here though.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.