Elasticsearch failed to start yesterday

Saplog · May 31, 2017, 8:12am

I'm using elasticserach since 2 month and yesterday, elasticsearch failed to start. I check log in /var/log/elasticsearch but thery are nothing....
Thanks for help

● elasticsearch.service - Elasticsearch
' Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: failed (Result: exit-code) since mer. 2017-05-31 11:52:35 CEST; 17min ago
Docs: http://www.elastic.co
Process: 4081 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 4079 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 4081 (code=exited, status=1/FAILURE)

mai 31 11:52:21 elkf systemd[1]: Started Elasticsearch.
mai 31 11:52:35 elkf systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
mai 31 11:52:35 elkf systemd[1]: Unit elasticsearch.service entered failed state.

Xavy · May 31, 2017, 8:18am

Hello:

Could you please check and share logs on /var/log/elasticsearch/* ?

Saplog · May 31, 2017, 8:22am

I just want to said that i have remove all logs files of elasticsearch, then i try to start again elasticsearch and this files appears.

root@elkf:/var/log/elasticsearch# ls -l /var/log/elasticsearch/*
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_access.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:52 /var/log/elasticsearch/elasticsearch.log

All of this file are empty.

Xavy · May 31, 2017, 8:27am

Hello @Saplog:

I would then suggest running:

/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR}

Changing variables to the appropriate value as stated on the systemctl config file

This should print an error on screen which lead to the root cause of your issue.

Saplog · May 31, 2017, 8:34am

/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR}

Okay i'll try , but what do you put on ${DATA_DIR} ?

/usr/share/elasticsearch/bin/elasticsearch -p 10278/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch/elasticsearch.log -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=/etc/elasticsearch/elasticsearch.yml

Xavy · May 31, 2017, 8:40am

Hello:

Please check /usr/lib/systemd/system/elasticsearch.service; for the variables values.

In my env test, they are:

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

but they could be different on your case (my test scenario is using repo packages for Ubuntu)

Saplog · May 31, 2017, 8:42am

Permission denied, but i'm log with root

Xavy · May 31, 2017, 8:43am

I meant that you opened such a file with a text editor (vi, nano, whatever) to check the variables values on it

Saplog · May 31, 2017, 8:45am

Oh yes sorry ,

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

Xavy · May 31, 2017, 8:47am

Then ${DATA_DIR} in your case should be /var/lib/elasticsearch. Please go on substituting each variable by its value as stated on the mentioned file, and then run the command I told you on comment #3

Saplog · May 31, 2017, 8:48am

It's done , and i have now that on /var/log/elasticsearch/elasticsearch.log

[2017-05-31T12:44:00,813][ERROR][o.e.b.Bootstrap ] Exception
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.1.jar:5.3.1]
[2017-05-31T12:44:00,884][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.3.1.jar:5.3.1]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.1.jar:5.3.1]
... 6 more

Xavy · May 31, 2017, 8:50am

Has such a command shown any kind of output, or written to ES log files?

Saplog · May 31, 2017, 8:53am

I dont understand what do you want to see

pts0 · May 31, 2017, 8:57am

log ist telling you the problem :

can not run elasticsearch as root

you should start as "normal" user, not root. Check the user is set in service script. You posted before

user= elasticsearch

Saplog · May 31, 2017, 9:00am

Okay but why it was workings before and now i can start him with roots
So i should add the user in the service script ?

Xavy · May 31, 2017, 9:05am

Hello:

I'm sorry my instructions were not as complete or as clear as it could be desirable.
I'll show a step by step diagnose:

From root: edit /etc/passwd so you assign elasticsearch user a shell (probably it will have /bin/false as shell)
Change to elasticsearch user: su - elasticsearch
Run /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elastisearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

This should bring to your screen something like:

[2017-05-31T10:57:43,480][INFO ][o.e.n.Node [2017-05-31T10:57:43,587][INFO ][o.e.e.NodeEnvironment [2017-05-31T10:57:43,587][INFO ][o.e.e.NodeEnvironment [2017-05-31T10:57:43,597][INFO ][o.e.n.Node [2017-05-31T10:57:43,599][INFO ][o.e.n.Node [2017-05-31T10:57:44,996][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,996][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,997][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,997][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService [2017-05-31T10:57:44,999][INFO ][o.e.p.PluginsService [2017-05-31T10:57:45,000][INFO ][o.e.p.PluginsService [2017-05-31T10:57:47,615][INFO ][o.e.d.DiscoveryModule [2017-05-31T10:57:48,663][INFO ][o.e.n.Node [2017-05-31T10:57:48,666][INFO ][o.e.n.Node [2017-05-31T10:57:48,894][INFO ][o.e.t.TransportService [2017-05-31T10:57:48,907][INFO ][o.e.b.BootstrapChecks ] [logs-test-01] initializing ...
] [logs-test-01] using [1] data paths, mounts [[/ (/dev/mapper/mov--test--01--vg-root)]], net usable_space [9.9gb], net total_space [13.1gb], spins? [possibly], types [ext4]
] [logs-test-01] heap size [1.9gb], compressed ordinary object pointers [true]
] [logs-test-01] node name [logs-test-01], node ID [2DrvmHHiSYSXrfoIcJQgyA]
] [logs-test-01] version[5.4.0], pid[767], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/4.4.0-62-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
] [logs-test-01] loaded module [aggs-matrix-stats]
] [logs-test-01] loaded module [ingest-common]
] [logs-test-01] loaded module [lang-expression]
] [logs-test-01] loaded module [lang-groovy]
] [logs-test-01] loaded module [lang-mustache]
] [logs-test-01] loaded module [lang-painless]
] [logs-test-01] loaded module [percolator]
] [logs-test-01] loaded module [reindex]
] [logs-test-01] loaded module [transport-netty3]
] [logs-test-01] loaded module [transport-netty4]
] [logs-test-01] no plugins loaded
] [logs-test-01] using discovery type [zen]
] [logs-test-01] initialized
] [logs-test-01] starting ...
] [logs-test-01] publish_address {192.168.160.204:9300}, bound_addresses {0.0.0.0:9300}
] [logs-test-01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks

Please share the output you receive when you run such a command, and remember to reedit the /etc/passwd to disable elasticsearch interactive sessions

Saplog · May 31, 2017, 9:11am

Okay , it's a memory problem right ?

[2017-05-31T13:05:41,398][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: Failed to create temporary file for /com/sun/jna/linux-x86/libjnidispatch.so library: Aucun espace disponible sur le périphérique
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:866) ~[jna-4.2.2.jar:4.2.2 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)]
at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_131]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_131]
at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.1.jar:5.3.1]
2017-05-31 13:05:41,416 main ERROR Unable to write to stream /var/log/elasticsearch/elasticsearch.log for appender rolling: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/elasticsearch/elasticsearch.log
2017-05-31 13:05:41,419 main ERROR An exception occurred processing Appender rolling org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/elasticsearch/elasticsearch.log
at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:254)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.writeToDestination(RollingFileManager.java:142)
at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:284)
at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:293)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:178)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:169)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:160)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:281)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:447)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:432)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:402)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
at org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:67)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1988)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1960)
at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2536)
at org.elasticsearch.bootstrap.Natives.(Natives.java:50)
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.Command.main(Command.java:88)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Caused by: java.io.IOException: Aucun espace disponible sur le périphérique
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:252)
... 34 more

Xavy · May 31, 2017, 9:14am

Hello:

It looks like a disk being full:
Failed to create temporary file for /com/sun/jna/linux-x86/libjnidispatch.so library: Aucun espace disponible sur le périphérique
[..]
Error writing to stream /var/log/elasticsearch/elasticsearch.log

Could you please share out put of df -h command?

Saplog · May 31, 2017, 9:19am

elasticsearch@elkf:/$ df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sda1           38G     38G     0 100% /
udev                10M       0   10M   0% /dev
tmpfs              810M     31M  779M   4% /run
tmpfs              2,0G       0  2,0G   0% /dev/shm
tmpfs              5,0M       0  5,0M   0% /run/lock
tmpfs              2,0G       0  2,0G   0% /sys/fs/cgroup

He is full

Xavy · May 31, 2017, 9:24am

Hello @Saplog:

Then you have the root cause of your issue. Make sure to make some room in your filesystem and ES will start to work again.
Please mark comment as solved if this helped you