Can't create lock file when sharing data.path folder between docker and host

[mathias]

My elasticsearch container works fine when using default data.path

I start it like this:
docker run -d --rm --net=host --name=elasticsearch elasticsearch2

But when I create a folder on my host machine and share this with the docker as data.path I fails to start:

docker run -d --rm --net=host --name=elasticsearch -v "$PWD/elasticsearch/data":/usr/share/elasticsearch/data elasticsearch2

[2017-04-07T13:36:05,396][INFO ][o.e.n.Node ] [] initializing ...
[2017-04-07T13:36:05,440][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.2.jar:5.2.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:260) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:239) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
... 6 more
Caused by: java.io.IOException: Mount point not found
at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]
at sun.nio.fs.UnixFileStore.(UnixFileStore.java:65) ~[?:?]
at sun.nio.fs.LinuxFileStore.(LinuxFileStore.java:44) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) ~[?:?]
at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_121]
at org.elasticsearch.env.ESFileStore.getMatchingFileStore(ESFileStore.java:107) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.env.Environment.getFileStore(Environment.java:340) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.env.NodeEnvironment$NodePath.(NodeEnvironment.java:108) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:227) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
... 6 more

What have I missed?

Br Mathias

[nik9000 (Nik Everett)]

Have a look at https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#max-local-storage-nodes

If you are doing this in production it probably is a good idea not to share data directories though.

[mathias]

Thanks for your reply Nik.

Unfortunately does not increasing node.max_local_storage_nodes change anything.

I tried with 2 and 10 but no difference.

WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
[2017-04-07T16:03:37,226][INFO ][o.e.n.Node ] [] initializing ...
[2017-04-07T16:03:37,280][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock ids [0--9]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [10])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.2.jar:5.2.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock ids [0--9]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [10])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:260) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.node.Node.(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/9

bash-4.3# ls -l /usr/share/elasticsearch/
total 224
-rw-r--r-- 1 elastics elastics 11358 Feb 24 17:24 LICENSE.txt
-rw-r--r-- 1 elastics elastics 172700 Feb 24 17:24 NOTICE.txt
-rw-r--r-- 1 elastics elastics 9108 Feb 24 17:24 README.textile
drwxr-xr-x 1 elastics elastics 4096 Apr 7 16:03 bin
drwxr-xr-x 1 elastics elastics 4096 Apr 7 16:03 config
drwxr-xr-x 3 elastics elastics 4096 Apr 7 15:59 data
drwxr-xr-x 1 elastics elastics 4096 Feb 24 17:29 lib
drwxr-xr-x 1 elastics elastics 4096 Apr 7 11:50 logs
drwxr-xr-x 1 elastics elastics 4096 Feb 24 17:29 modules
drwxr-xr-x 1 elastics elastics 4096 Feb 24 17:29 plugins
bash-4.3# ls -l /usr/share/elasticsearch/data
total 4
drwxr-xr-x 12 elastics elastics 4096 Apr 7 16:03 nodes
bash-4.3# ls -l /usr/share/elasticsearch/data/nodes/
total 40
drwxr-xr-x 2 elastics elastics 4096 Apr 7 15:59 0
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:01 1
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 2
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 3
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 4
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 5
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 6
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 7
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 8
drwxr-xr-x 2 elastics elastics 4096 Apr 7 16:03 9
bash-4.3# ls -l /usr/share/elasticsearch/data/nodes/0/
total 0
-rw-r--r-- 1 elastics elastics 0 Apr 7 15:59 node.lock
bash-4.3# ls -l /usr/share/elasticsearch/data/nodes/1/
total 0
-rw-r--r-- 1 elastics elastics 0 Apr 7 16:01 node.lock
bash-4.3# ls -l /usr/share/elasticsearch/data/nodes/2/
total 0
-rw-r--r-- 1 elastics elastics 0 Apr 7 16:03 node.lock

Must be something else that is wrong.

I am running the docker in a vm and need access to a larger disk on a cinder volume which I can have from docker host. That is the reason why I need to share the data folder with docker host.

Br Mathias

[jasontedor]

@mathias

Caused by: java.io.IOException: Mount point not found
atsun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]

This is the root of your problem. What storage driver are you using? Is it either of overlay or overlay2 by chance? Are you using a musl libc based image?

[mathias]

Hi Json,

Here are some more background details:
I used Alpine docker Elasticsearch v5.2.2 running on ubuntu16.04 (cloud image).
Cinder volume is not connected to my cloud so I currently just want to store my data in home directory of ubuntu.

sblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sr0 iso9660 config-2 2017-04-07-13-37-53-00
vda
└─vda1 ext4 cloudimg-rootfs b14f53da-7bc9-4604-a014-341f669f87c0 /

blkid
/dev/vda1: LABEL="cloudimg-rootfs" UUID="b14f53da-7bc9-4604-a014-341f669f87c0" TYPE="ext4" PARTUUID="f1917857-01"
/dev/sr0: UUID="2017-04-07-13-37-53-00" LABEL="config-2" TYPE="iso9660"

uname -a
Linux docker-vm-0 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@docker-vm-0:~/Dockers/elasticsearch2$ cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

I start my mongodb docker in the same way and I do not have any problem whatsoever whit in this case.

Br Mathias

[jasontedor]

Thanks, that confirms that you're using musl libc. Unless I'm overlooking something, I do not see enough information to conclude whether or not you are using either of overlay or overlay2. I strongly suspect that you are. In this case, I think that you are a victim of this bug. If you switch your storage driver away from overlay or overlay2 (say, aufs), or to an image based on glibc, you should be fine.

[mathias]

Thanks for that Jason,

Overlay2 seems to be what I am using.

sudo ls -l /var/lib/docker/
total 64
drwx------ 16 root root 4096 Apr 8 05:13 containers
drwx------ 3 root root 4096 Apr 7 11:41 image
drwxr-x--- 3 root root 4096 Apr 7 11:41 network
drwx------ 269 root root 28672 Apr 8 05:13 overlay2
drwx------ 4 root root 4096 Apr 7 11:41 plugins
drwx------ 2 root root 4096 Apr 7 11:41 swarm
drwx------ 2 root root 4096 Apr 8 05:03 tmp
drwx------ 2 root root 4096 Apr 7 11:41 trust
drwx------ 7 root root 4096 Apr 8 05:13 volumes
ubuntu@docker-vm-0:~$ sudo ls -l /var/lib/docker/overlay2
total 1088
drwx------ 5 root root 4096 Apr 7 11:47 018b82cd15470d1ee0cba21290ec84c8f2c3d75985815d7d2a81499585f08cd5
drwx------ 5 root root 4096 Apr 7 11:46 04c22cc4052327a70c4b95a16daff0fea0e3a14a5943bbf908a1643fb9e58e87
drwx------ 5 root root 4096 Apr 7 19:13 05e8f3b217c83a32e3070e6b39ca66dd3360ff77d6fe121d032a387ee09ae351

I will monitor the bug and hope for a quick fix.

Thanks
Mathias

[jasontedor]

You're welcome.

[mathias]

Hi,

I can also confirm that the debian based image works well.

Br Mathias