Elasticsearch failed to restart

Elastic Stack Elasticsearch
1

Hi,

Please, I need help with this !!

[root@frghcslnetv12 elasticsearch]# systemctl restart elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

[root@frghcslnetv12 elasticsearch]# journalctl -xe
-- Subject: Unit collector-sidecar.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit collector-sidecar.service has finished starting up.
--
-- The start-up result is done.
Jul 13 15:16:32 frghcslnetv12 systemd[1]: Starting Wrapper service for Graylog controlled collector...
-- Subject: Unit collector-sidecar.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit collector-sidecar.service has begun starting up.
Jul 13 15:16:32 frghcslnetv12 graylog-collector-sidecar[58094]: time="2018-07-13T15:16:32+02:00" level=fatal msg="Unable to open configuration file: /etc/graylog/collector-sidec
Jul 13 15:16:32 frghcslnetv12 systemd[1]: collector-sidecar.service: main process exited, code=exited, status=1/FAILURE
Jul 13 15:16:32 frghcslnetv12 systemd[1]: Unit collector-sidecar.service entered failed state.
Jul 13 15:16:32 frghcslnetv12 systemd[1]: collector-sidecar.service failed.
Jul 13 15:18:32 frghcslnetv12 systemd[1]: collector-sidecar.service holdoff time over, scheduling restart.
Jul 13 15:18:32 frghcslnetv12 systemd[1]: Started Wrapper service for Graylog controlled collector.
-- Subject: Unit collector-sidecar.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit collector-sidecar.service has finished starting up.
--
-- The start-up result is done.
Jul 13 15:18:32 frghcslnetv12 systemd[1]: Starting Wrapper service for Graylog controlled collector...
-- Subject: Unit collector-sidecar.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit collector-sidecar.service has begun starting up.
Jul 13 15:18:32 frghcslnetv12 graylog-collector-sidecar[58212]: time="2018-07-13T15:18:32+02:00" level=fatal msg="Unable to open configuration file: /etc/graylog/collector-sidec
Jul 13 15:18:32 frghcslnetv12 systemd[1]: collector-sidecar.service: main process exited, code=exited, status=1/FAILURE
Jul 13 15:18:32 frghcslnetv12 systemd[1]: Unit collector-sidecar.service entered failed state.
Jul 13 15:18:32 frghcslnetv12 systemd[1]: collector-sidecar.service failed.
Jul 13 15:20:01 frghcslnetv12 systemd[1]: Started Session 4305 of user root.
-- Subject: Unit session-4305.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-4305.scope has finished starting up.
--
-- The start-up result is done.
Jul 13 15:20:01 frghcslnetv12 systemd[1]: Starting Session 4305 of user root.
-- Subject: Unit session-4305.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-4305.scope has begun starting up.

[root@frghcslnetv12 elasticsearch]# systemctl status elasticsearch.service                                                                                                       ● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2018-07-13 14:53:58 CEST; 27min ago
     Docs: http://www.elastic.co
  Process: 55201 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=143)
  Process: 56723 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=203/EXEC)
 Main PID: 55201 (code=exited, status=143)

Jul 13 14:53:58 frghcslnetv12 systemd[1]: Starting Elasticsearch...
Jul 13 14:53:58 frghcslnetv12 systemd[56723]: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such file or directory
Jul 13 14:53:58 frghcslnetv12 systemd[1]: elasticsearch.service: control process exited, code=exited status=203
Jul 13 14:53:58 frghcslnetv12 systemd[1]: Failed to start Elasticsearch.
Jul 13 14:53:58 frghcslnetv12 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 13 14:53:58 frghcslnetv12 systemd[1]: elasticsearch.service failed.
2

Jul 13 14:53:58 frghcslnetv12 systemd[56723]: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such file or directory

Does the elasticsearch-systemd-pre-exec file exist in the /usr/share/elasticsearch/bin/ directory?

3

No, it doesn't exist !! :confused:

4

Remove it from your unit file or init script.

5

where ?

6

systemctl edit --full elasticsearch and remove the line that is starting with /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec.

7

I don't understand, it starts and turns down just after :

[root@frghcslnetv12 bin]# systemctl start elasticsearch.service
[root@frghcslnetv12 bin]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-07-13 16:11:17 CEST; 1s ago
     Docs: http://www.elastic.co
 Main PID: 61658 (java)
    Tasks: 17
   CGroup: /system.slice/elasticsearch.service
           └─61658 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava....

Jul 13 16:11:17 frghcslnetv12 systemd[1]: Started Elasticsearch.
Jul 13 16:11:17 frghcslnetv12 systemd[1]: Starting Elasticsearch...
[root@frghcslnetv12 bin]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2018-07-13 16:11:19 CEST; 1s ago
     Docs: http://www.elastic.co
  Process: 61658 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
 Main PID: 61658 (code=exited, status=1/FAILURE)

Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,360 main ERROR Null object returned for RollingFile in Appenders.
Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,361 main ERROR Null object returned for RollingFile in Appenders.
Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,361 main ERROR Unable to locate appender "rolling" for logger config "root"
Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,361 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config ...log.index"
Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,362 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearc...uditTrail"
Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,362 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "i...h.slowlog"
Jul 13 16:11:19 frghcslnetv12 elasticsearch[61658]: 2018-07-13 16:11:19,362 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasti...precation"
Jul 13 16:11:19 frghcslnetv12 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 13 16:11:19 frghcslnetv12 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 13 16:11:19 frghcslnetv12 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
8

In your elasticsearch.yml file in the /etc/elasticsearch directory, what do you have set for the path.logs? Make sure the location specified for the path.logs is owned by elasticsearch.

1 Like
9

Yes, it is owned by elasticsearch :

[root@frghcslnetv12 bin]# cd /data
[root@frghcslnetv12 data]# ls
elasticsearch
[root@frghcslnetv12 data]# ls -l
total 4
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Jun 19 11:41 elasticsearch
1 Like
10

Are any log files be created in that directory?

11

No, they are old !!

[root@frghcslnetv12 indices]# ls -l
total 8
drwxr-xr-x 7 elasticsearch elasticsearch 4096 Jul  4 16:56 S7Pj25k1Rj6bark5lKr-OQ
drwxr-xr-x 7 elasticsearch elasticsearch 4096 Jul  4 16:56 zbybtLYVTFOnh5U6b4sDgg
[root@frghcslnetv12 indices]#
12

Those are actually your elasticsearch indices not your logs. I believe your in the wrong path.

13 
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
14

in /var/log/elasticsearch :

[root@frghcslnetv12 elasticsearch]# ls -l
total 13936
-rw-r--r-- 1 elasticsearch elasticsearch       0 Jun 18 10:56 elasticsearch_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch       0 Jun 18 10:56 elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch       0 Jun 18 10:56 elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch   21493 Jun 18 11:53 elasticsearch.log
-rw-r--r-- 1 elasticsearch elasticsearch    2921 Jul 13 16:35 gc.log.0.current
-rw-r--r-- 1 elasticsearch elasticsearch 7234477 Jun 18 23:59 my-application-2018-06-18.log
-rw-r--r-- 1 elasticsearch elasticsearch       0 Jun 18 12:00 my-application_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch       0 Jun 18 12:00 my-application_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch       0 Jun 18 12:00 my-application_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 6997905 Jun 19 10:43 my-application.log
15

That explains it. Uncomment out the path.logs and set it to /var/log/elasticsearch

16

I did it but, I still have the same problem :

[root@frghcslnetv12 elasticsearch]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-07-13 16:42:46 CEST; 9s ago
     Docs: http://www.elastic.co
 Main PID: 64204 (java)
    Tasks: 22
   CGroup: /system.slice/elasticsearch.service
           ├─64204 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava....
           └─64268 /usr/share/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller

Jul 13 16:42:46 frghcslnetv12 systemd[1]: Started Elasticsearch.
Jul 13 16:42:46 frghcslnetv12 systemd[1]: Starting Elasticsearch...
[root@frghcslnetv12 elasticsearch]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2018-07-13 16:42:55 CEST; 697ms ago
     Docs: http://www.elastic.co
  Process: 64204 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
 Main PID: 64204 (code=exited, status=1/FAILURE)

Jul 13 16:42:46 frghcslnetv12 systemd[1]: Started Elasticsearch.
Jul 13 16:42:46 frghcslnetv12 systemd[1]: Starting Elasticsearch...
Jul 13 16:42:55 frghcslnetv12 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 13 16:42:55 frghcslnetv12 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 13 16:42:55 frghcslnetv12 systemd[1]: elasticsearch.service failed.
17

There are logs now :

[root@frghcslnetv12 elasticsearch]# tail -30 network-logs.log
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:135) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
                at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
        Suppressed: java.lang.IllegalArgumentException: unknown setting [default.path.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:344) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:308) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:282) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:135) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
                at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
18

Are the logs being updated now? If they are, can you show what errors are being logged?

19

I did it, do you need mode ? !!

20

Yeah, more would be great.