Error start elasticsearch in centos7

Rizky_Hudha · July 12, 2021, 8:50am

hello i found a problem when starting elasticsearch, and an error like this :

Jul 12 05:05:34 bigdata systemd-entrypoint[22275]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp
Jul 12 05:05:34 bigdata systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 12 05:05:34 bigdata systemd[1]: Failed to start Elasticsearch.
Jul 12 05:05:34 bigdata systemd[1]: Unit elasticsearch.service entered failed state.
Jul 12 05:05:34 bigdata systemd[1]: elasticsearch.service failed.

and this tail -f /var/log/elasticsearch/elasticsearch.log

[2021-07-12T03:15:14,441][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [bigdata] [controller/26724] [Main.cc@169] ML controller exiting
[2021-07-12T03:15:14,443][INFO ][o.e.x.m.p.NativeController] [bigdata] Native controller process has stopped - no new native processes can be started
[2021-07-12T03:15:14,620][INFO ][o.e.c.s.ClusterSettings  ] [bigdata] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2021-07-12T03:15:14,882][INFO ][o.e.l.LicenseService     ] [bigdata] license [1996a97d-c8bc-491c-a001-b0339a62d490] mode [basic] - valid
[2021-07-12T03:15:14,884][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [bigdata] Active license is now [BASIC]; Security is disabled
[2021-07-12T03:15:14,884][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [bigdata] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.13/security-minimal-setup.html to enable security.
[2021-07-12T03:15:14,889][INFO ][o.e.g.GatewayService     ] [bigdata] recovered [40] indices into cluster_state
[2021-07-12T03:15:14,903][INFO ][o.e.n.Node               ] [bigdata] stopped
[2021-07-12T03:15:14,904][INFO ][o.e.n.Node               ] [bigdata] closing ...
[2021-07-12T03:15:14,915][INFO ][o.e.n.Node               ] [bigdata] closed

how to solve the error above?

spinscale · July 12, 2021, 8:54am

can you share the full log instead of the above snippet? Nothing jumps out to me.

For testing purposes, can you disable machine learning via xpack.ml.enabled in the configuration and try again?

Rizky_Hudha · July 12, 2021, 9:49am

this is full error log

Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at java.base/java.nio.file.Files.newOutputStream(Files.java:224)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:410)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:406)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:254)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:484)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:253)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:228)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:343)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.cli.Command.main(Command.java:79)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Jul 12 09:38:14 bigdata systemd-entrypoint[22633]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Jul 12 09:38:14 bigdata systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 12 09:38:14 bigdata systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit elasticsearch.service has failed.
-- 
-- The result is failed.
Jul 12 09:38:14 bigdata systemd[1]: Unit elasticsearch.service entered failed state.
Jul 12 09:38:14 bigdata systemd[1]: elasticsearch.service failed.
Jul 12 09:38:14 bigdata polkitd[622]: Unregistered Authentication Agent for unix-process:22627:39434988 (system bus name :1.357, object path /org/freedesktop/PolicyKit1/
Jul 12 09:46:02 bigdata polkitd[622]: Registered Authentication Agent for unix-process:22835:39482037 (system bus name :1.358 [/usr/bin/pkttyagent --notify-fd 5 --fallba
Jul 12 09:46:02 bigdata systemd[1]: Started Kibana.
-- Subject: Unit kibana.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kibana.service has finished starting up.
-- 
-- The start-up result is done.
Jul 12 09:46:02 bigdata polkitd[622]: Unregistered Authentication Agent for unix-process:22835:39482037 (system bus name :1.358, object path /org/freedesktop/PolicyKit1/
lines 1529-1566/1566 (END)

spinscale · July 12, 2021, 11:13am

There must be more of that stack trace earlier in the log, that is needed as well..

Rizky_Hudha · July 13, 2021, 2:13pm

this is /var/log/elasticsearch/elasticsearch.log

	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.3.jar:7.13.3]
		at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.3.jar:7.13.3]
		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.3.jar:7.13.3]
		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.3.jar:7.13.3]

and this error from journalctl -u elasticsearch

Jul 13 08:53:53 bigdata systemd[1]: Starting Elasticsearch...
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: org.elasticsearch.cli.UserException: unable 
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at java.base/java.nio.file.Files.newOutputStream(Files.java:224)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:410)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:406)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:254)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:484)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:253)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:228)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:343)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.cli.Command.main(Command.java:79)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Jul 13 08:54:02 bigdata systemd-entrypoint[1403]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Jul 13 08:54:03 bigdata systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 13 08:54:03 bigdata systemd[1]: Failed to start Elasticsearch.
Jul 13 08:54:03 bigdata systemd[1]: Unit elasticsearch.service entered failed state.
Jul 13 08:54:03 bigdata systemd[1]: elasticsearch.service failed.
Jul 13 09:57:43 bigdata systemd[1]: Starting Elasticsearch...
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: org.elasticsearch.cli.UserException: unable
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at java.base/java.nio.file.Files.newOutputStream(Files.java:224)
Jul 13 09:57:47 bigdata systemd-entrypoint[10474]: at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:410)

spinscale · July 13, 2021, 2:18pm

Can you share the permissions of /etc/elasticsearch/elasticsearch.keystore - if that is owned by root and not by the elasticsearch group that might be the issue (and also needs to be readable by the elasticsearch group).

What Elasticsearch version is this?

Rizky_Hudha · July 13, 2021, 2:56pm

[root@bigdata elasticsearch]# ll
total 164
-rw-rw---- 1 root elasticsearch   199 Jul 13 14:07 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch  2801 Jul 13 14:43 elasticsearch.yml
-rw-r--r-- 1 root           989 23111 Jun 11 14:35 hs_err_pid24963.log
-rw-r--r-- 1 root           989 23126 Jun 11 14:37 hs_err_pid25558.log
-rw-r--r-- 1 root           989 23077 Jun 11 14:39 hs_err_pid25688.log
-rw-r--r-- 1 root           989 23077 Jun 11 14:40 hs_err_pid26090.log
-rw-r--r-- 1 root           989 23105 Jun  9 13:58 hs_err_pid8475.log
-rw-rw---- 1 root elasticsearch  3182 Jul  2 12:13 jvm.options
drwxr-s--- 2 root elasticsearch     6 Jul  2 12:17 jvm.options.d
-rw-rw---- 1 root           989  3180 Jul 12 04:28 jvm.options.rpmsave
-rw-rw---- 1 root elasticsearch 18626 Jul  2 12:13 log4j2.properties
-rw-rw---- 1 root elasticsearch   473 Jul  2 12:13 role_mapping.yml
-rw-rw---- 1 root elasticsearch   197 Jul  2 12:13 roles.yml
-rw-rw---- 1 root elasticsearch     0 Jul  2 12:13 users
-rw-rw---- 1 root elasticsearch     0 Jul  2 12:13 users_roles

i use version 7.13.2

spinscale · July 13, 2021, 2:59pm

can you also share the permissions of the /etc/elasticsearch directory itself? Thank you!

Rizky_Hudha · July 13, 2021, 3:09pm

you mean something like this

[root@bigdata elasticsearch]# pwd
/etc/elasticsearch
[root@bigdata elasticsearch]# ll -h
total 164K
-rw-rw---- 1 root elasticsearch  199 Jul 13 14:07 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch 2,8K Jul 13 14:43 elasticsearch.yml
-rw-r--r-- 1 root           989  23K Jun 11 14:35 hs_err_pid24963.log
-rw-r--r-- 1 root           989  23K Jun 11 14:37 hs_err_pid25558.log
-rw-r--r-- 1 root           989  23K Jun 11 14:39 hs_err_pid25688.log
-rw-r--r-- 1 root           989  23K Jun 11 14:40 hs_err_pid26090.log
-rw-r--r-- 1 root           989  23K Jun  9 13:58 hs_err_pid8475.log
-rw-rw---- 1 root elasticsearch 3,2K Jul  2 12:13 jvm.options
drwxr-s--- 2 root elasticsearch    6 Jul  2 12:17 jvm.options.d
-rw-rw---- 1 root           989 3,2K Jul 12 04:28 jvm.options.rpmsave
-rw-rw---- 1 root elasticsearch  19K Jul  2 12:13 log4j2.properties
-rw-rw---- 1 root elasticsearch  473 Jul  2 12:13 role_mapping.yml
-rw-rw---- 1 root elasticsearch  197 Jul  2 12:13 roles.yml
-rw-rw---- 1 root elasticsearch    0 Jul  2 12:13 users
-rw-rw---- 1 root elasticsearch    0 Jul  2 12:13 users_roles

spinscale · July 14, 2021, 8:29am

No more like ls -ld /etc/elasticsearch

system · August 11, 2021, 8:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unit elasticsearch.service has failed to start on centos 7 Elasticsearch	11	1959	January 2, 2022
Not able to start elasticsearch service Elasticsearch	5	3112	October 9, 2020
Elasticsearch 7.13 Elasticsearch elastic-stack-security	26	1909	November 4, 2022
XPack Security Enabled TLS/SSL configured, still not working! Elasticsearch elastic-stack-security	5	1956	January 13, 2021
Elasticsearch installation and configuration issue Elasticsearch	25	6368	September 30, 2020

Error start elasticsearch in centos7

Related topics