/var/lib/elasticsearch/tmp not accessible in ES 6.8

Chris_Gatihi · October 20, 2020, 5:43pm

Hi,

I'm doing an upgrade from ES 5.6 to ES 6.8. When starting ES 6.8, I'm getting this exception:

    -- Logs begin at Tue 2020-10-20 00:29:07 UTC, end at Tue 2020-10-20 17:35:18 UTC. --
    Oct 20 00:29:56  systemd[1]: Started Elasticsearch.
    Oct 20 00:30:02  systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
    Oct 20 00:30:02  systemd[1]: Unit elasticsearch.service entered failed state.
    Oct 20 00:30:02  systemd[1]: elasticsearch.service failed.
    Oct 20 06:03:54  systemd[1]: Started Elasticsearch.
    Oct 20 06:03:55  elasticsearch[117234]: ERROR: Temporary file directory [/var/lib/elasticsearch/tmp] does not exist or is not accessible
    Oct 20 06:03:55  systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
    Oct 20 06:03:55  systemd[1]: Unit elasticsearch.service entered failed state.
    Oct 20 06:03:55  systemd[1]: elasticsearch.service failed.
    Oct 20 06:07:43  systemd[1]: Started Elasticsearch.
    Oct 20 06:07:45  elasticsearch[118278]: 2020-10-20 06:07:45,919 main ERROR Unable to locate appender "deprecated_audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.DeprecatedLoggingAuditTrail"
    Oct 20 06:07:46  elasticsearch[118278]: Oct 20, 2020 6:07:46 AM com.sun.jna.Native loadNativeDispatchLibrary
    Oct 20 06:07:46  elasticsearch[118278]: WARNING: JNA Warning: IOException removing temporary files
    Oct 20 06:07:46  elasticsearch[118278]: java.io.IOException: JNA temporary directory '/var/lib/elasticsearch/tmp' is not writable
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.getTempDir(Native.java:1310)
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.removeTemporaryFiles(Native.java:1317)
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:919)
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.<clinit>(Native.java:195)
    Oct 20 06:07:46  elasticsearch[118278]: at java.lang.Class.forName0(Native Method)
    Oct 20 06:07:46  elasticsearch[118278]: at java.lang.Class.forName(Class.java:264)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.cli.Command.main(Command.java:90)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
    Oct 20 06:07:46  elasticsearch[118278]: [2020-10-20T06:07:46,388][WARN ][o.e.b.Natives            ] [unknown] unable to load JNA native support library, native methods will be disabled.
    Oct 20 06:07:46  elasticsearch[118278]: java.lang.UnsatisfiedLinkError: Failed to create temporary file for /com/sun/jna/linux-x86-64/libjnidispatch.so library: JNA temporary directory '/var/lib/elasticsearch/tmp' is not writable
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032) ~[jna-5.5.0.jar:5.5.0 (b0)]
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) ~[jna-5.5.0.jar:5.5.0 (b0)]
    Oct 20 06:07:46  elasticsearch[118278]: at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.5.0.jar:5.5.0 (b0)]
    Oct 20 06:07:46  elasticsearch[118278]: at java.lang.Class.forName0(Native Method) ~[?:1.8.0_262]
    Oct 20 06:07:46  elasticsearch[118278]: at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_262]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) [elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.12.jar:6.8.12]
    ...
    Oct 20 06:07:46  elasticsearch[118278]: ... 6 more
    Oct 20 06:07:46  elasticsearch[118278]: Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/tmp
    Oct 20 06:07:46  elasticsearch[118278]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_262]
    Oct 20 06:07:46  elasticsearch[118278]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_262]
    Oct 20 06:07:46  elasticsearch[118278]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_262]
    Oct 20 06:07:46  elasticsearch[118278]: at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308) ~[?:1.8.0_262]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:409) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:291) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:254) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:207) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.12.jar:6.8.12]
    Oct 20 06:07:46  elasticsearch[118278]: ... 6 more
    Oct 20 06:07:46  systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
    Oct 20 06:07:46  systemd[1]: Unit elasticsearch.service entered failed state.
    Oct 20 06:07:46  systemd[1]: elasticsearch.service failed.

If I create the /var/lib/elasticsearch/tmp directory and give it the needed permissions, ES starts fine. But I didn't have to do this in 5.6 (in 5.6, either /var/lib/elasticsearch/tmp was automatically created or ES didn't try to write to it). Is this now a necessary preparation prior to starting ES 6.8?

Thanks,
Chris

spinscale · October 21, 2020, 8:11am

Can you share who own /var/lib/elasticsearch/ by running ls -ld /var/lib/elasticsearch/ - can you make sure that this is owned by the user running elasticsearch (likely elasticsearch).

Chris_Gatihi · October 21, 2020, 5:41pm

Thanks for following up, @spinscale.

Yes, that directory is owned by elasticsearch user who is starting ES, which is why I was scratching my head.

After googling around a bit, I found a couple of references to the following commands issued prior to starting ES:

mkdir -p /var/lib/elasticsearch/tmp
chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/tmp

I went ahead and added this and ES starts fine. Just wasn't entirely clear to me if this is indeed an official requirement or not.

Thanks,
Chris

system · November 18, 2020, 5:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.