/etc/init.d/elasticsearch fails at system boot

Elastic Stack Elasticsearch
1

Hi,

I installed elasticsearch 1.1.1 from rpm package on redhat 6.2

Init script /etc/init.d/elasticsearch is installed and run at server boot
but *no java process is started or it dies silently. No log in
/var/log/elasticsearch/ but the lock file is touched
(/var/lock/subsys/elasticsearch)

If I run (as root) "/etc/init.d/elasticsearch start", the node is started.

I added a logger statement to /usr/share/elasticsearch/bin/elasticsearch to
see what is actually run. Here is what is written to /var/log/message :

exec "/usr/bin/java" -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true -XX:+
UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -
XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.
path.home="/usr/share/elasticsearch"
-cp
":/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/"
-Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/
log/elasticsearch
-Des.default.path.data=/var/product/elasticsearch -Des.default.path.work=
/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

If I run manually this command line after switching to elasticsearch user,
the node is started.

In summary, everything works normally except that elasticsearch won't start
automatically after a server reboot.

Any clue ? I'm stuck

--
Zac

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I had trouble with this a while ago and in my case it turned out to be that
I didn't have enough ram to allocate and the JDK was crashing. That
probably isn't what is up with yours but you can follow along with the
issue starting here:

These kinds of problems irk me because I don't really have a good way to
solve them. Maybe pipe the stdout/stderr of the Elasticsearch process in
the init script to tmp temporarily file then try to start it. The JDK has
a habit of printing errors to stderr because, when it fails, it doesn't
have anywhere else to send them.

Path problems, maybe?

Nik

On Tue, May 6, 2014 at 9:12 AM, Zacharie Elcor zelcor@gmail.com wrote:

Hi,

I installed elasticsearch 1.1.1 from rpm package on redhat 6.2

Init script /etc/init.d/elasticsearch is installed and run at server boot
but *no java process is started or it dies silently. No log in
/var/log/elasticsearch/ but the lock file is touched
(/var/lock/subsys/elasticsearch)

If I run (as root) "/etc/init.d/elasticsearch start", the node is started.

I added a logger statement to /usr/share/elasticsearch/bin/elasticsearch
to see what is actually run. Here is what is written to /var/log/message :

exec "/usr/bin/java" -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true -XX
:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -
Des.path.home="/usr/share/elasticsearch"
-cp
":/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/"
-Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=
/var/log/elasticsearch
-Des.default.path.data=/var/product/elasticsearch -Des.default.path.work=
/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

If I run manually this command line after switching to elasticsearch user,
the node is started.

In summary, everything works normally except that elasticsearch won't
start automatically after a server reboot.

Any clue ? I'm stuck

--
Zac

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YdtC3tJtvhhuizRLSgJE8qTh8aUW%2Bz9_a-daEWmd%3DkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Many thanks Nik. Not a RAM issue but your second advice was helpful.

I added "> /tmp/logfile 2>&1" at the end of line "daemon --user..." in
/etc/init.d/elasticsearch.

After rebbot, /tmp/logfile contained a message saying that env var HOSTNAME
was not defined. In my case, HOSTNAME is used in elasticsearch.yml to
define node name.

So the rule seems to be : If you make use of environment variables in
elasticsearch.yml (as explained in
Elasticsearch Platform — Find real-time answers at scale | Elastic),
make sure these var are defined when the service starts.

--
Zac

On Tuesday, May 6, 2014 3:24:55 PM UTC+2, Nikolas Everett wrote:

I had trouble with this a while ago and in my case it turned out to be
that I didn't have enough ram to allocate and the JDK was crashing. That
probably isn't what is up with yours but you can follow along with the
issue starting here:
Init script should allow the user to investigate stdout and stderr if initializing fails · Issue #5234 · elastic/elasticsearch · GitHub

These kinds of problems irk me because I don't really have a good way to
solve them. Maybe pipe the stdout/stderr of the Elasticsearch process in
the init script to tmp temporarily file then try to start it. The JDK has
a habit of printing errors to stderr because, when it fails, it doesn't
have anywhere else to send them.

Path problems, maybe?

Nik

On Tue, May 6, 2014 at 9:12 AM, Zacharie Elcor <zel...@gmail.com<javascript:>

wrote:

Hi,

I installed elasticsearch 1.1.1 from rpm package on redhat 6.2

Init script /etc/init.d/elasticsearch is installed and run at server boot
but *no java process is started or it dies silently. No log in
/var/log/elasticsearch/ but the lock file is touched
(/var/lock/subsys/elasticsearch)

If I run (as root) "/etc/init.d/elasticsearch start", the node is started.

I added a logger statement to /usr/share/elasticsearch/bin/elasticsearch
to see what is actually run. Here is what is written to /var/log/message :

exec "/usr/bin/java" -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true -XX
:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -
Des.path.home="/usr/share/elasticsearch"
-cp
":/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/"
-Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=
/var/log/elasticsearch
-Des.default.path.data=/var/product/elasticsearch -Des.default.path.work
=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

If I run manually this command line after switching to elasticsearch
user, the node is started.

In summary, everything works normally except that elasticsearch won't
start automatically after a server reboot.

Any clue ? I'm stuck

--
Zac

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/960cd129-0226-4395-9a2e-64f19113baba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

What is the actual variable you used in your yml?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 7 May 2014 19:54, Zacharie Elcor zelcor@gmail.com wrote:

Many thanks Nik. Not a RAM issue but your second advice was helpful.

I added "> /tmp/logfile 2>&1" at the end of line "daemon --user..." in
/etc/init.d/elasticsearch.

After rebbot, /tmp/logfile contained a message saying that env var
HOSTNAME was not defined. In my case, HOSTNAME is used in elasticsearch.yml
to define node name.

So the rule seems to be : If you make use of environment variables in
elasticsearch.yml (as explained in
Elasticsearch Platform — Find real-time answers at scale | Elastic),
make sure these var are defined when the service starts.

--
Zac

On Tuesday, May 6, 2014 3:24:55 PM UTC+2, Nikolas Everett wrote:

I had trouble with this a while ago and in my case it turned out to be
that I didn't have enough ram to allocate and the JDK was crashing. That
probably isn't what is up with yours but you can follow along with the
issue starting here: GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine
issues/5234

These kinds of problems irk me because I don't really have a good way to
solve them. Maybe pipe the stdout/stderr of the Elasticsearch process in
the init script to tmp temporarily file then try to start it. The JDK has
a habit of printing errors to stderr because, when it fails, it doesn't
have anywhere else to send them.

Path problems, maybe?

Nik

On Tue, May 6, 2014 at 9:12 AM, Zacharie Elcor zel...@gmail.com wrote:

Hi,

I installed elasticsearch 1.1.1 from rpm package on redhat 6.2

Init script /etc/init.d/elasticsearch is installed and run at server
boot but *no java process is started or it dies silently. No log in
/var/log/elasticsearch/ but the lock file is touched (/var/lock/subsys/
elasticsearch)

If I run (as root) "/etc/init.d/elasticsearch start", the node is
started.

I added a logger statement to /usr/share/elasticsearch/bin/elasticsearch
to see what is actually run. Here is what is written to /var/log/message :

exec "/usr/bin/java" -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true -
XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancy
Only -XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -
Des.path.home="/usr/share/elasticsearch"
-cp ":/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/
usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/" -
Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=
/var/log/elasticsearch
-Des.default.path.data=/var/product/elasticsearch -Des.default.path.
work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

If I run manually this command line after switching to elasticsearch
user, the node is started.

In summary, everything works normally except that elasticsearch won't
start automatically after a server reboot.

Any clue ? I'm stuck

--
Zac

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/1b462c05-d03a-4283-b8a4-b84f37f01cc2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/960cd129-0226-4395-9a2e-64f19113baba%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/960cd129-0226-4395-9a2e-64f19113baba%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aDbKauX%3DBDY6QWNcs5EAH%3DZ6Z8e9NC_W5qykezMY_qng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

node.name: node.${HOSTNAME}

On Wednesday, May 7, 2014 12:01:17 PM UTC+2, Mark Walkom wrote:

What is the actual variable you used in your yml?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/38746dda-0c16-48a0-9cc8-ed6bfd70230e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.