In our local development environments we use docker-compose to run Elasticsearch and Kibana. We're getting ready to upgrade from 5.6 to 6.3, so my starting point was updating our docker-compose.yml file to point to the newer Docker images.
However, when I run docker-compose up Elasticsearch fails to start with the following error:
Recreating webcrawl_elasticsearch_1 ... done
Attaching to webcrawl_elasticsearch_1
elasticsearch_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1 | [2018-07-12T16:32:26,826][INFO ][o.e.n.Node ] [] initializing ...
elasticsearch_1 | [2018-07-12T16:32:26,850][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
elasticsearch_1 | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:243) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | ... 6 more
elasticsearch_1 | Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:222) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | ... 6 more
elasticsearch_1 | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
elasticsearch_1 | at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elasticsearch_1 | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elasticsearch_1 | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elasticsearch_1 | at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:178) ~[?:?]
elasticsearch_1 | at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
elasticsearch_1 | at java.nio.channels.FileChannel.open(FileChannel.java:340) ~[?:?]
elasticsearch_1 | at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
elasticsearch_1 | at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
elasticsearch_1 | at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:208) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1 | ... 6 more
If I switch back to the 5.6 Elasticsearch Docker image, Elasticsearch runs fine.
Here is our docker-compose configuration:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
command: "elasticsearch -Escript.engine.groovy.inline.search=true -Escript.engine.groovy.inline.aggs=true -Ehttp.cors.enabled=true -Ehttp.cors.allow-origin='*'"
volumes:
- elasticsearch:/usr/share/elasticsearch/data
ports:
- 9210:9200
kibana:
image: docker.elastic.co/elasticsearch/kibana:6.3.1
links:
- elasticsearch
ports:
- 5611:5601