Elasticsearch start failed when Path.data change in elasticsearch.yml

phongddh · September 20, 2018, 4:31am

I setup the elasticsearch 6.4.1 on CentOS machine and its works fine until i uncomment the "path.data: /var/www/STO-44/elasticsearch/" setting in elasticsearch.yml .

Info:

[root@elk-fshare ~]# ll /var/www/STO-44/
total 4
drwxr-x--- 3 elasticsearch elasticsearch 25 Sep 20 10:44 elasticsearch

Error:

[root@elk-fshare ~]# systemctl status elasticsearch -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2018-09-20 11:14:23 +07; 14min ago
Docs: http://www.elastic.co
Process: 1848 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ?${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 1848 (code=exited, status=1/FAILURE)
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,666 main ERROR Null ?object returned for RollingFile in Appenders.
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,667 main ERROR Null object returned for RollingFile in Appenders.
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,667 main ERROR Unable to locate appender "rolling" for logger config "root"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,667 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,668 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,668 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Sep 20 11:14:23 elk-fshare elasticsearch[1848]: 2018-09-20 11:14:23,668 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Sep 20 11:14:23 elk-fshare systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Sep 20 11:14:23 elk-fshare systemd[1]: Unit elasticsearch.service entered failed state.
Sep 20 11:14:23 elk-fshare systemd[1]: elasticsearch.service failed.

warkolm · September 20, 2018, 4:34am

What does the actual Elasticsearch log show, from /var/log/elasticsearch/.

phongddh · September 20, 2018, 4:46am

Dear Mark Walkom,

Thank you for reply,

Log show form /var/log/elasticsearch/gc.log.0.current

tailf /var/log/elasticsearch/gc.log.0.current

Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 65771592k(62566752k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=357916672 -XX:MaxTenuringThreshold=6 -XX:NewSize=357916672 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=715825152 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2018-09-20T11:44:37.918+0700: 0.364: Total time for which application threads were stopped: 0.0001491 seconds, Stopping threads took: 0.0000384 seconds
2018-09-20T11:44:37.946+0700: 0.393: Total time for which application threads were stopped: 0.0000716 seconds, Stopping threads took: 0.0000171 seconds
2018-09-20T11:44:38.015+0700: 0.462: Total time for which application threads were stopped: 0.0000868 seconds, Stopping threads took: 0.0000186 seconds
2018-09-20T11:44:38.243+0700: 0.690: Total time for which application threads were stopped: 0.0001994 seconds, Stopping threads took: 0.0000275 seconds
2018-09-20T11:44:38.244+0700: 0.691: Total time for which application threads were stopped: 0.0000579 seconds, Stopping threads took: 0.0000118 seconds
2018-09-20T11:44:38.256+0700: 0.703: Total time for which application threads were stopped: 0.0000895 seconds, Stopping threads took: 0.0000170 seconds
2018-09-20T11:44:38.264+0700: 0.711: Total time for which application threads were stopped: 0.0000837 seconds, Stopping threads took: 0.0000152 seconds
2018-09-20T11:44:38.277+0700: 0.724: Total time for which application threads were stopped: 0.0000879 seconds, Stopping threads took: 0.0000190 seconds
2018-09-20T11:44:38.351+0700: 0.798: Total time for which application threads were stopped: 0.0001411 seconds, Stopping threads took: 0.0000226 seconds
2018-09-20T11:44:38.544+0700: 0.991: Total time for which application threads were stopped: 0.0002858 seconds, Stopping threads took: 0.0000272 seconds
2018-09-20T11:44:38.615+0700: 1.062: Total time for which application threads were stopped: 0.0002988 seconds, Stopping threads took: 0.0000437 seconds
2018-09-20T11:44:38.891+0700: 1.337: Total time for which application threads were stopped: 0.0003663 seconds, Stopping threads took: 0.0000337 seconds
2018-09-20T11:44:38.911+0700: 1.358: Total time for which application threads were stopped: 0.0002602 seconds, Stopping threads took: 0.0000235 seconds
2018-09-20T11:44:38.923+0700: 1.370: Total time for which application threads were stopped: 0.0001059 seconds, Stopping threads took: 0.0000137 seconds
Heap
par new generation total 314560K, used 139820K [0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
eden space 279616K, 50% used [0x00000000c0000000, 0x00000000c888b110, 0x00000000d1110000)
from space 34944K, 0% used [0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
to space 34944K, 0% used [0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
concurrent mark-sweep generation total 699072K, used 0K [0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 17289K, capacity 17788K, committed 17920K, reserved 1064960K
class space used 2086K, capacity 2270K, committed 2304K, reserved 1048576K

Br,

phongddh · September 20, 2018, 4:50am

And, in /var/log/elasticsearch/elasticsearch.log

[2018-09-20T11:48:46,852][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-09-20T11:48:46,854][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2018-09-20T11:48:46,855][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-09-20T11:48:46,855][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2018-09-20T11:48:46,855][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-09-20T11:48:46,908][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:298) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_181]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_181]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_181]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:412) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.4.1.jar:6.4.1]
... 12 more
[2018-09-20T11:48:46,914][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:298) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_181]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_181]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_181]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_181]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_181]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:412) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:298) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-6.4.1.jar:6.4.1]

warkolm · September 20, 2018, 9:31pm

That's why, did you change the permissions of this path at all? What are the current permissions?

system · October 18, 2018, 9:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.