Elasticsearch won't start after upgrade from 5.6.8 to 6.0.1. Failed to read local state - directory not empty

Elastic Stack Elasticsearch
1

Hi,

I'm trying to upgrade our stack from 5.6.8 to 6.0.1 on an Ubuntu OS using the deb file after running Upgrade Assistant on 5.6.8. I'm seeing the following error in /var/log/elasticsearch/elasticsearch.log:

2018-03-01T10:07:59,682][INFO ][o.e.c.u.IndexFolderUpgrader] [hydra-2017.12.28/0xWw9wcuQgG9LQ1cfCwf6A] upgrading [/resilient/elasticsearch/nodes/0/indices/hydra-2017.12.28] to new naming convention
[2018-03-01T10:07:59,688][ERROR][o.e.g.GatewayMetaState   ] [L4o6lDe] failed to read local state, exiting...
java.nio.file.FileSystemException: /resilient/elasticsearch/nodes/0/indices/hydra-2017.12.28 -> /resilient/elasticsearch/nodes/0/indices/0xWw9wcuQgG9LQ1cfCwf6A: Directory not empty
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) ~[?:?]
	at java.nio.file.Files.move(Files.java:1395) ~[?:1.8.0_151]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:65) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:112) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:424) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:246) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]

The errors cascade from there. I have already checked the service file at /usr/lib/systemd/system/elasticsearch.service as mentioned in some other threads and everything looks good. I have path.data set to /resilient/elasticsearch in etc/elasticsearch/elasticsearch.yml. FWIW I originally upgraded this stack from 2.x to 5.6.8 (after running migration plugin).

I even then tried just going to 6.2.2 and still had this same error.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.