Upgrade with docker-compose fails to start

Elastic Stack Elasticsearch
1

In our local development environments we use docker-compose to run Elasticsearch and Kibana. We're getting ready to upgrade from 5.6 to 6.3, so my starting point was updating our docker-compose.yml file to point to the newer Docker images.

However, when I run docker-compose up Elasticsearch fails to start with the following error:

Recreating webcrawl_elasticsearch_1 ... done
Attaching to webcrawl_elasticsearch_1
elasticsearch_1  | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1  | [2018-07-12T16:32:26,826][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2018-07-12T16:32:26,850][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
elasticsearch_1  | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
elasticsearch_1  | 	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:243) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	... 6 more
elasticsearch_1  | Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
elasticsearch_1  | 	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:222) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	... 6 more
elasticsearch_1  | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
elasticsearch_1  | 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elasticsearch_1  | 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elasticsearch_1  | 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elasticsearch_1  | 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:178) ~[?:?]
elasticsearch_1  | 	at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
elasticsearch_1  | 	at java.nio.channels.FileChannel.open(FileChannel.java:340) ~[?:?]
elasticsearch_1  | 	at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
elasticsearch_1  | 	at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
elasticsearch_1  | 	at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
elasticsearch_1  | 	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:208) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.1.jar:6.3.1]
elasticsearch_1  | 	... 6 more

If I switch back to the 5.6 Elasticsearch Docker image, Elasticsearch runs fine.

Here is our docker-compose configuration:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
    command: "elasticsearch -Escript.engine.groovy.inline.search=true -Escript.engine.groovy.inline.aggs=true -Ehttp.cors.enabled=true -Ehttp.cors.allow-origin='*'"
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9210:9200

  kibana:
    image: docker.elastic.co/elasticsearch/kibana:6.3.1
    links:
      - elasticsearch
    ports:
      - 5611:5601
2

Solved. Turns out we were using the docker images from hub.docker.com before and the issue was permissions in the disk volume. The solution was to mount the volume in the elasticsearch container but run a shell instead of the default and change the permissions for /usr/share/elasticsearch/data

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.