Elasticsearch failed to start yesterday

I'm using elasticserach since 2 month and yesterday, elasticsearch failed to start. I check log in /var/log/elasticsearch but thery are nothing....
Thanks for help :slight_smile:

● elasticsearch.service - Elasticsearch

' Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: failed (Result: exit-code) since mer. 2017-05-31 11:52:35 CEST; 17min ago
Docs: http://www.elastic.co
Process: 4081 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 4079 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 4081 (code=exited, status=1/FAILURE)

mai 31 11:52:21 elkf systemd[1]: Started Elasticsearch.
mai 31 11:52:35 elkf systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
mai 31 11:52:35 elkf systemd[1]: Unit elasticsearch.service entered failed state.

Hello:

Could you please check and share logs on /var/log/elasticsearch/* ?

1 Like

I just want to said that i have remove all logs files of elasticsearch, then i try to start again elasticsearch and this files appears.

root@elkf:/var/log/elasticsearch# ls -l /var/log/elasticsearch/*
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_access.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:49 /var/log/elasticsearch/elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 mai 31 11:52 /var/log/elasticsearch/elasticsearch.log

All of this file are empty.

Hello @Saplog:

I would then suggest running:

/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR}

Changing variables to the appropriate value as stated on the systemctl config file

This should print an error on screen which lead to the root cause of your issue.

1 Like

/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR}

Okay i'll try , but what do you put on ${DATA_DIR} ?

/usr/share/elasticsearch/bin/elasticsearch -p 10278/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch/elasticsearch.log -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=/etc/elasticsearch/elasticsearch.yml

Hello:

Please check /usr/lib/systemd/system/elasticsearch.service; for the variables values.

In my env test, they are:

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

but they could be different on your case (my test scenario is using repo packages for Ubuntu)

1 Like

Permission denied, but i'm log with root :confused:

I meant that you opened such a file with a text editor (vi, nano, whatever) to check the variables values on it

1 Like

Oh yes sorry ,

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

Then ${DATA_DIR} in your case should be /var/lib/elasticsearch. Please go on substituting each variable by its value as stated on the mentioned file, and then run the command I told you on comment #3

1 Like

It's done , and i have now that on /var/log/elasticsearch/elasticsearch.log

[2017-05-31T12:44:00,813][ERROR][o.e.b.Bootstrap ] Exception
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.1.jar:5.3.1]
[2017-05-31T12:44:00,884][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.3.1.jar:5.3.1]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.1.jar:5.3.1]
... 6 more

Has such a command shown any kind of output, or written to ES log files?

1 Like

I dont understand what do you want to see :confused:

log ist telling you the problem :

can not run elasticsearch as root

you should start as "normal" user, not root. Check the user is set in service script. You posted before

user= elasticsearch

1 Like

Okay but why it was workings before and now i can start him with roots :expressionless:
So i should add the user in the service script ?

Hello:

I'm sorry my instructions were not as complete or as clear as it could be desirable.
I'll show a step by step diagnose:

  1. From root: edit /etc/passwd so you assign elasticsearch user a shell (probably it will have /bin/false as shell)
  2. Change to elasticsearch user: su - elasticsearch
  3. Run /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elastisearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

This should bring to your screen something like:

[2017-05-31T10:57:43,480][INFO ][o.e.n.Node ] [logs-test-01] initializing ...
[2017-05-31T10:57:43,587][INFO ][o.e.e.NodeEnvironment ] [logs-test-01] using [1] data paths, mounts [[/ (/dev/mapper/mov--test--01--vg-root)]], net usable_space [9.9gb], net total_space [13.1gb], spins? [possibly], types [ext4]
[2017-05-31T10:57:43,587][INFO ][o.e.e.NodeEnvironment ] [logs-test-01] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-05-31T10:57:43,597][INFO ][o.e.n.Node ] [logs-test-01] node name [logs-test-01], node ID [2DrvmHHiSYSXrfoIcJQgyA]
[2017-05-31T10:57:43,599][INFO ][o.e.n.Node ] [logs-test-01] version[5.4.0], pid[767], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/4.4.0-62-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-05-31T10:57:44,996][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [aggs-matrix-stats]
[2017-05-31T10:57:44,996][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [ingest-common]
[2017-05-31T10:57:44,997][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [lang-expression]
[2017-05-31T10:57:44,997][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [lang-groovy]
[2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [lang-mustache]
[2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [lang-painless]
[2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [percolator]
[2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [reindex]
[2017-05-31T10:57:44,998][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [transport-netty3]
[2017-05-31T10:57:44,999][INFO ][o.e.p.PluginsService ] [logs-test-01] loaded module [transport-netty4]
[2017-05-31T10:57:45,000][INFO ][o.e.p.PluginsService ] [logs-test-01] no plugins loaded
[2017-05-31T10:57:47,615][INFO ][o.e.d.DiscoveryModule ] [logs-test-01] using discovery type [zen]
[2017-05-31T10:57:48,663][INFO ][o.e.n.Node ] [logs-test-01] initialized
[2017-05-31T10:57:48,666][INFO ][o.e.n.Node ] [logs-test-01] starting ...
[2017-05-31T10:57:48,894][INFO ][o.e.t.TransportService ] [logs-test-01] publish_address {192.168.160.204:9300}, bound_addresses {0.0.0.0:9300}
[2017-05-31T10:57:48,907][INFO ][o.e.b.BootstrapChecks ] [logs-test-01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks

Please share the output you receive when you run such a command, and remember to reedit the /etc/passwd to disable elasticsearch interactive sessions

1 Like

Okay , it's a memory problem right ?

[2017-05-31T13:05:41,398][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: Failed to create temporary file for /com/sun/jna/linux-x86/libjnidispatch.so library: Aucun espace disponible sur le périphérique
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:866) ~[jna-4.2.2.jar:4.2.2 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)]
at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_131]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_131]
at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.1.jar:5.3.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.1.jar:5.3.1]
2017-05-31 13:05:41,416 main ERROR Unable to write to stream /var/log/elasticsearch/elasticsearch.log for appender rolling: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/elasticsearch/elasticsearch.log
2017-05-31 13:05:41,419 main ERROR An exception occurred processing Appender rolling org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/elasticsearch/elasticsearch.log
at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:254)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.writeToDestination(RollingFileManager.java:142)
at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:284)
at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:293)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:178)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:169)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:160)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:281)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:447)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:432)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:402)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
at org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:67)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1988)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1960)
at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2536)
at org.elasticsearch.bootstrap.Natives.(Natives.java:50)
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.Command.main(Command.java:88)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Caused by: java.io.IOException: Aucun espace disponible sur le périphérique
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:252)
... 34 more

Hello:

It looks like a disk being full:
Failed to create temporary file for /com/sun/jna/linux-x86/libjnidispatch.so library: Aucun espace disponible sur le périphérique
[..]
Error writing to stream /var/log/elasticsearch/elasticsearch.log

Could you please share out put of df -h command?

1 Like
elasticsearch@elkf:/$ df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sda1           38G     38G     0 100% /
udev                10M       0   10M   0% /dev
tmpfs              810M     31M  779M   4% /run
tmpfs              2,0G       0  2,0G   0% /dev/shm
tmpfs              5,0M       0  5,0M   0% /run/lock
tmpfs              2,0G       0  2,0G   0% /sys/fs/cgroup

He is full :cry:

Hello @Saplog:

Then you have the root cause of your issue. Make sure to make some room in your filesystem and ES will start to work again. :slight_smile:
Please mark comment as solved if this helped you

1 Like