Elasticsearch start failed when Path.data change in elasticsearch.yml

phongddh · September 20, 2018, 4:31am

I setup the elasticsearch 6.4.1 on CentOS machine and its works fine until i uncomment the "path.data: /var/www/STO-44/elasticsearch/" setting in elasticsearch.yml .

Info:

[root@elk-fshare ~]# ll /var/www/STO-44/
total 4
drwxr-x--- 3 elasticsearch elasticsearch 25 Sep 20 10:44 elasticsearch

Error:

[root@elk-fshare ~]# systemctl status elasticsearch -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2018-09-20 11:14:23 +07; 14min ago
Docs: http://www.elastic.co
Process: 1848 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ?${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 1848 (code=exited, status=1/FAILURE)
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,666 main ERROR Null ?object returned for RollingFile in Appenders.
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,667 main ERROR Null object returned for RollingFile in Appenders.
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,667 main ERROR Unable to locate appender "rolling" for logger config "root"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,667 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,668 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,668 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,668 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Sep 20 11:14:23 elk-fshare systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Sep 20 11:14:23 elk-fshare systemd[1]: Unit elasticsearch.service entered failed state.
Sep 20 11:14:23 elk-fshare systemd[1]: elasticsearch.service failed.

warkolm · September 20, 2018, 4:34am

What does the actual Elasticsearch log show, from /var/log/elasticsearch/.

phongddh · September 20, 2018, 4:46am

Dear Mark Walkom,

Thank you for reply,

Log show form /var/log/elasticsearch/gc.log.0.current

tailf /var/log/elasticsearch/gc.log.0.current

Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 65771592k(62566752k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=357916672 -XX:MaxTenuringThreshold=6 -XX:NewSize=357916672 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=715825152 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2018-09-20T11:44:37.918+0700: 0.364: Total time for which application threads were stopped: 0.0001491 seconds, Stopping threads took: 0.0000384 seconds
2018-09-20T11:44:37.946+0700: 0.393: Total time for which application threads were stopped: 0.0000716 seconds, Stopping threads took: 0.0000171 seconds
2018-09-20T11:44:38.015+0700: 0.462: Total time for which application threads were stopped: 0.0000868 seconds, Stopping threads took: 0.0000186 seconds
2018-09-20T11:44:38.243+0700: 0.690: Total time for which application threads were stopped: 0.0001994 seconds, Stopping threads took: 0.0000275 seconds
2018-09-20T11:44:38.244+0700: 0.691: Total time for which application threads were stopped: 0.0000579 seconds, Stopping threads took: 0.0000118 seconds
2018-09-20T11:44:38.256+0700: 0.703: Total time for which application threads were stopped: 0.0000895 seconds, Stopping threads took: 0.0000170 seconds
2018-09-20T11:44:38.264+0700: 0.711: Total time for which application threads were stopped: 0.0000837 seconds, Stopping threads took: 0.0000152 seconds
2018-09-20T11:44:38.277+0700: 0.724: Total time for which application threads were stopped: 0.0000879 seconds, Stopping threads took: 0.0000190 seconds
2018-09-20T11:44:38.351+0700: 0.798: Total time for which application threads were stopped: 0.0001411 seconds, Stopping threads took: 0.0000226 seconds
2018-09-20T11:44:38.544+0700: 0.991: Total time for which application threads were stopped: 0.0002858 seconds, Stopping threads took: 0.0000272 seconds
2018-09-20T11:44:38.615+0700: 1.062: Total time for which application threads were stopped: 0.0002988 seconds, Stopping threads took: 0.0000437 seconds
2018-09-20T11:44:38.891+0700: 1.337: Total time for which application threads were stopped: 0.0003663 seconds, Stopping threads took: 0.0000337 seconds
2018-09-20T11:44:38.911+0700: 1.358: Total time for which application threads were stopped: 0.0002602 seconds, Stopping threads took: 0.0000235 seconds
2018-09-20T11:44:38.923+0700: 1.370: Total time for which application threads were stopped: 0.0001059 seconds, Stopping threads took: 0.0000137 seconds
Heap
par new generation total 314560K, used 139820K [0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
eden space 279616K, 50% used [0x00000000c0000000, 0x00000000c888b110, 0x00000000d1110000)
from space 34944K, 0% used [0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
to space 34944K, 0% used [0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
concurrent mark-sweep generation total 699072K, used 0K [0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 17289K, capacity 17788K, committed 17920K, reserved 1064960K
class space used 2086K, capacity 2270K, committed 2304K, reserved 1048576K

Br,

phongddh · September 20, 2018, 4:50am

And, in /var/log/elasticsearch/elasticsearch.log

[2018-09-20T11:48:46,852][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-09-20T11:48:46,854][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2018-09-20T11:48:46,855][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-09-20T11:48:46,855][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2018-09-20T11:48:46,855][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-09-20T11:48:46,908][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:298) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_181]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_181]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_181]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:412) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.4.1.jar:6.4.1]
... 12 more
[2018-09-20T11:48:46,914][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:298) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_181]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_181]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_181]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:412) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:298) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-6.4.1.jar:6.4.1]

warkolm · September 20, 2018, 9:31pm

That's why, did you change the permissions of this path at all? What are the current permissions?

system · October 18, 2018, 9:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ElasticSearch won't start when path.logs and path.data is changed Elasticsearch	2	7759	July 5, 2017
Elasticsearch failed on start when Path.data change in elasticsearch.yml Elasticsearch	3	1681	July 17, 2017
Unable to start elasticsearch on CentOs Elasticsearch	8	40402	July 5, 2017
Error encountered after modified elasticsearch.yml Elasticsearch	19	8153	April 25, 2018
Elasticsearch failed after few seconds of start Elasticsearch	1	579	September 18, 2018

Elasticsearch start failed when Path.data change in elasticsearch.yml

Related Topics