Elasticsearch won't start after upgrade from 5.6.8 to 6.0.1. Failed to read local state - directory not empty

Hi,

I'm trying to upgrade our stack from 5.6.8 to 6.0.1 on an Ubuntu OS using the deb file after running Upgrade Assistant on 5.6.8. I'm seeing the following error in /var/log/elasticsearch/elasticsearch.log:

2018-03-01T10:07:59,682][INFO ][o.e.c.u.IndexFolderUpgrader] [hydra-2017.12.28/0xWw9wcuQgG9LQ1cfCwf6A] upgrading [/resilient/elasticsearch/nodes/0/indices/hydra-2017.12.28] to new naming convention
[2018-03-01T10:07:59,688][ERROR][o.e.g.GatewayMetaState   ] [L4o6lDe] failed to read local state, exiting...
java.nio.file.FileSystemException: /resilient/elasticsearch/nodes/0/indices/hydra-2017.12.28 -> /resilient/elasticsearch/nodes/0/indices/0xWw9wcuQgG9LQ1cfCwf6A: Directory not empty
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) ~[?:?]
	at java.nio.file.Files.move(Files.java:1395) ~[?:1.8.0_151]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:65) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:112) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:424) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:246) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]

The errors cascade from there. I have already checked the service file at /usr/lib/systemd/system/elasticsearch.service as mentioned in some other threads and everything looks good. I have path.data set to /resilient/elasticsearch in etc/elasticsearch/elasticsearch.yml. FWIW I originally upgraded this stack from 2.x to 5.6.8 (after running migration plugin).

I even then tried just going to 6.2.2 and still had this same error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.