# Elasticserach bootstrap fail if elasticsearch.keystore.tmp already exists **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [June 15, 2020, 8:06am UTC](https://discuss.elastic.co/t/elasticserach-bootstrap-fail-if-elasticsearch-keystore-tmp-already-exists/237087 "2020-06-15T08:06:09Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![gab.bernasconi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gab.bernasconi/32/57955_2.png) [@gab.bernasconi](https://discuss.elastic.co/u/gab.bernasconi)\ **Post date:** [June 15, 2020, 8:06am UTC](https://discuss.elastic.co/t/elasticserach-bootstrap-fail-if-elasticsearch-keystore-tmp-already-exists/237087/1 "2020-06-15T08:06:09Z")
**Environment** - Elasticsearch version: 7.4.2 (docker) **Problem Summary** If Elasticsearch already finds the file `/usr/share/elasticsearch/config/elasticsearch.keystore.tmp` during bootstrap checks, it fails to start **Steps to reproduce (PRODUCTION)** Start the Elasticsearch docker container, and stop it when it is creating **elasticsearch.keystore** (this is very difficult to do manually, but it's exacly what happened in our case) **Steps to reproduce (SIMULATION)** Create a custom docker-compose file where you map a volume that already contains an empty `elasticsearch.keystore.tmp` file mapped to `/usr/share/elasticsearch/config/elasticsearch.keystore.tmp`, like the following: ```auto version: '3.7' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.4.2 environment: - node.name=es01 - cluster.name=es-sample-cluster - discovery.type=single-node - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms4g -Xmx4g" ulimits: memlock: soft: -1 hard: -1 volumes: - ./elasticsearch.keystore.tmp:/usr/share/elasticsearch/config/elasticsearch.keystore.tmp ports: - 9200:9200 ``` **Expected Result** Elasticsearch starts correctly **Actual Result** Elasticsearch does not start, and fails with the following error: ```auto elasticsearch_1 | Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.FileAlreadyExistsException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp elasticsearch_1 | Likely root cause: java.nio.file.FileAlreadyExistsException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp elasticsearch_1 | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) elasticsearch_1 | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) elasticsearch_1 | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) elasticsearch_1 | at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) elasticsearch_1 | at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478) elasticsearch_1 | at java.base/java.nio.file.Files.newOutputStream(Files.java:223) elasticsearch_1 | at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:410) elasticsearch_1 | at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:406) elasticsearch_1 | at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:254) elasticsearch_1 | at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:484) elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:242) elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:305) elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:90) elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) elasticsearch_1 | Refer to the log for complete error details. das_elasticsearch_1 exited with code 1 ``` **Considerations** I narrowed down the problem to the following method: [https://github.com/elastic/elasticsearch/blob/v7.4.2/server/src/main/java/org/elasticsearch/common/settings/KeyStoreWrapper.java#L478](https://github.com/elastic/elasticsearch/blob/v7.4.2/server/src/main/java/org/elasticsearch/common/settings/KeyStoreWrapper.java#L478) The `KeyStoreWrapper.save()` method does not explicitly handle `java.nio.file.FileAlreadyExistsException`, so it simply fails and exits. I would expect that this exception would be handled explicitly, logged, and would allow the service to start all the same... ---
**Author:** ![gab.bernasconi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gab.bernasconi/32/57955_2.png) [@gab.bernasconi](https://discuss.elastic.co/u/gab.bernasconi)\ **Post date:** [June 23, 2020, 7:55am UTC](https://discuss.elastic.co/t/elasticserach-bootstrap-fail-if-elasticsearch-keystore-tmp-already-exists/237087/2 "2020-06-23T07:55:20Z")
Bump... can anyone please advise on this? I'd like to know if I can open a bug on Github for this, or if there's a specific reason why the `elasticsearch.keystore` bootstrap check fails if the temporary file is already present. ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [June 23, 2020, 3:10pm UTC](https://discuss.elastic.co/t/elasticserach-bootstrap-fail-if-elasticsearch-keystore-tmp-already-exists/237087/3 "2020-06-23T15:10:47Z")
Is there any specific reason that you want/need to have an `elasticsearch.keystore.tmp` file in your mapped volume ? It would be a bug if elasticsearch leaves this lying there and you can't say with certainty that > this exception would be handled explicitly, logged, and would allow the service to start all the same... as depending on why this temporary file was left there, it might or might not be the right decision to log and overwrite this. ---
**Author:** ![gab.bernasconi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gab.bernasconi/32/57955_2.png) [@gab.bernasconi](https://discuss.elastic.co/u/gab.bernasconi)\ **Post date:** [June 23, 2020, 3:28pm UTC](https://discuss.elastic.co/t/elasticserach-bootstrap-fail-if-elasticsearch-keystore-tmp-already-exists/237087/4 "2020-06-23T15:28:23Z")
I do not need to map `elasticsearch.keystore.tmp` in production, it's only a way to replicate the problem. The situation I had in production was described in my previous post: > [@gab.bernasconi](#): > > **Steps to reproduce (PRODUCTION)** > Start the Elasticsearch docker container, and stop it when it is creating **elasticsearch.keystore** (this is very difficult to do manually, but it's exacly what happened in our case) The problem happens if the docker container is _stopped_ (not removed) exactly when `KeyStoreWrapper` is creating `elasticsearch.keystore.tmp`, but didn't rename it to `elasticsearch.keystore`. If this happens, when the container is restarted, it finds the `elasticsearch.keystore.tmp` in the folder, and crashes with an error. This condition is hard to replicate, because you have a window of a few milliseconds to stop the container exactly at the right moment. The workaround, in my case, is just to _stop and remove_ the container (i.e. `docker-compose down`) and then re-create (i.e. `docker-compose up`). Nonetheless, in our product we would like to avoid this, and - if possible - have Elasticsearch to be robust against the `elasticsearch.keystore.tmp` file being already there. Is there any specific reason why the `FileAlreadyExistsException` does not need to be handled explicitly? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 21, 2020, 3:28pm UTC](https://discuss.elastic.co/t/elasticserach-bootstrap-fail-if-elasticsearch-keystore-tmp-already-exists/237087/5 "2020-07-21T15:28:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.