Elasticsearch Keystore error: Device or Resource Busy while upgrading

Elastic Stack Elasticsearch
1

Hello,

I have a multi node Elasticsearch cluster running on version 8.13.4.
The cluster needed Slack Integration and AD/LDAP integration so it also has elasticsearch.keystore file bind mount to it. The cluster is set up in a dockerized environment using docker compose.

While upgrading cluster to 8.14.2 we are facing the following error

e1      | Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
e1      | 	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
e1      | 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
e1      | 	at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:882)
e1      | 	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
e1      | 	at java.base/java.nio.file.Files.move(Files.java:1430)
e1      | 	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:519)
e1      | 	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:433)
e1      | 	at org.elasticsearch.common.settings.KeyStoreWrapper.upgrade(KeyStoreWrapper.java:307)
e1      | 	at org.elasticsearch.common.settings.KeyStoreWrapper.bootstrap(KeyStoreWrapper.java:237)
e1      | 	at org.elasticsearch.server.cli.KeyStoreLoader.bootstrap(KeyStoreLoader.java:37)
e1      | 	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:88)
e1      | 	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
e1      | 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
e1      | 	at org.elasticsearch.cli.Command.main(Command.java:50)
e1      | 	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
e1 exited with code 1

It seems that the issue is related to bind mounting the elasticsearch.keystore file. This issue is mentioned here

According to this if we mount the config directory directly it also asks for other config files as well , that we want to avoid.

What should be the workaround for this issue ?

2

I have the exact same issue. I am trying to upgrade my cluster from 8.13.2 to 8.17.2 but I get the same problem with the .tmp file

I dont want to mount the entire config folder. That would not make sense I would have to manage all files and directories in that folder locally. That is why I chose to use docker containers. Now for every upgrade I should first extract the contents of the config folder from the image before I can even start an upgrade ?

This is clearly a bug or a very bad design. It would be better if the keystore file was in its own folder like config/keystore/ so I could just bind mount that folder and not the entire config folder.