Elastic search refuses to start after update

Elastic Stack Elasticsearch
1

So downgraded my Elasticsearch to 7.17 for me to achieve the all in one deployment with wazuh which was i have been using all along until someone updated the server and in then update the server. I then regenerated the keystore and even used debian to install another elasticsearch 7.17 using debian. but now Elasticsearch is refusing to start. this is the error am getting from logs

*[2025-01-20T09:57:15,961][ERROR][o.e.b.Bootstrap ] [node-1] Exception
org.elasticsearch.ElasticsearchException: failed to bind service

  • at org.elasticsearch.node.Node.(Node.java:1089) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:309) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) [elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) [elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) [elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.17.13.jar:7.17.13]*
    Caused by: java.io.IOException: failed to test writes in data directory [/var/lib/elasticsearch/nodes/0/indices/4acQvuK3QquuBDtX4aMNlA/0/index] write permission is required
  • at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1318) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1295) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:342) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:429) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • ... 11 more*
    Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/4acQvuK3QquuBDtX4aMNlA/0/index/.es_temp_file: Not a directory
  • at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]*
  • at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]*
  • at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]*
  • at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:291) ~[?:?]*
  • at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109) ~[?:?]*
  • at java.nio.file.Files.deleteIfExists(Files.java:1190) ~[?:?]*
  • at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1314) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1295) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:342) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:429) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • ... 11 more*
    [2025-01-20T09:57:15,967][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to test writes in data directory [/var/lib/elasticsearch/nodes/0/indices/4acQvuK3QquuBDtX4aMNlA/0/index] write permission is required]; nested: FileSystemException[/var/lib/elasticsearch/nodes/0/indices/4acQvuK3QquuBDtX4aMNlA/0/index/.es_temp_file: Not a directory];
  • at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.13.jar:7.17.13]*
    Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
  • at org.elasticsearch.node.Node.(Node.java:1089) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:309) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • ... 6 more*
    Caused by: java.io.IOException: failed to test writes in data directory [/var/lib/elasticsearch/nodes/0/indices/4acQvuK3QquuBDtX4aMNlA/0/index] write permission is required
  • at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1318) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1295) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:342) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:429) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:309) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • ... 6 more*
    Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/4acQvuK3QquuBDtX4aMNlA/0/index/.es_temp_file: Not a directory
  • at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]*
  • at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]*
  • at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]*
  • at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:291) ~[?:?]*
  • at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109) ~[?:?]*
  • at java.nio.file.Files.deleteIfExists(Files.java:1190) ~[?:?]*
  • at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1314) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1295) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:342) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:429) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.node.Node.(Node.java:309) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.13.jar:7.17.13]*
  • ... 6 more **
2

Elasticsearch does not support in place downgrades so you can not go from a higher version to a lower version. If you need to do this you need to spin up a new cluster and restore from a snapshot taken from the old version or possibly reindex from remote into the older version cluster from the new one.

2 Likes