"Device or resource busy" when adding to keystore in Docker Compose

I want to persist the elasticsearch.keystore file in a Docker Compose project.

Obviously there is a Catch-22 in that I could mount an existing keystore file, but I don't have one yet if I start with a new setup.

So first, I run this Compose-file via docker-compose -f docker-compose.setup.yml up:

version: '3.7'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: 7.6.2
    volumes:
      - type: bind
        source: ./elasticsearch/config/
        target: /config
      - type: bind
        source: ./elasticsearch/container_setup.sh
        target: /usr/share/elasticsearch/container_setup.sh
    command: ./container_setup.sh
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      ELASTIC_PASSWORD: ${ELASTICSEARCH_ELASTIC_PASSWORD:-changeme}

The container_setup.sh file is as follows:

#!//usr/bin/env bash

echo "=== CREATE Keystore ==="
echo "Elastic password is: $ELASTIC_PASSWORD"

if [ -f /config/elasticsearch/elasticsearch.keystore ]; then
    echo "Remove old elasticsearch.keystore from host config"
    rm /config/elasticsearch/elasticsearch.keystore
fi

echo "Creating keystore"
if [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]]; then
    echo "Keystore already exists"
else
    echo "Creating new keystore"
    /usr/share/elasticsearch/bin/elasticsearch-keystore create
fi

echo "Setting bootstrap.password..."
echo "$ELASTIC_PASSWORD" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x 'bootstrap.password'

echo "Moving keystore"
mv /usr/share/elasticsearch/config/elasticsearch.keystore /config/elasticsearch.keystore

What this does is create a new elasticsearch.keystore file in config on my host. Interestingly it will be owned by root:root:

➜ ll elasticsearch/config 
total 8
-rw-rw---- 1 root   root   243 Apr 15 14:53 elasticsearch.keystore
-rw-r--r-- 1 werner werner 963 Dec 11 18:25 elasticsearch.yml

Now I start the "real" Elastic service with the following Compose file:

version: '3.7'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: 7.6.2
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
      - type: bind
        source: ./elasticsearch/config/elasticsearch.keystore
        target: /usr/share/elasticsearch/config/elasticsearch.keystore
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      ELASTIC_PASSWORD: ${ELASTICSEARCH_ELASTIC_PASSWORD:-changeme}
    restart: unless-stopped

volumes:
    elasticsearch: {}

This works fine. The real issue occurs when I now try to add an entry to the keystore. So I run:

➜ docker-compose run --rm elasticsearch /bin/bash

And I try to add the file:

bin/elasticsearch-keystore add-file "gcs.client.foo.credentials_file" /tmp/auth.json

What I get is the following:

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/e
lasticsearch.keystore: Device or resource busy
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
        at java.base/java.nio.file.Files.move(Files.java:1425)
        at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:523)
        at org.elasticsearch.common.settings.AddFileKeyStoreCommand.execute(AddFileKeyStoreCommand.java:94)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)

The permissions in the container config directory are as follows before running that command:

[root@44dff2478ddc elasticsearch]# ls -lha config/
total 44K
drwxrwxr-x 1 elasticsearch root          4.0K Apr 15 12:58 .
drwxrwxr-x 1 elasticsearch root          4.0K Feb 29 00:20 ..
-rw-rw---- 1 root          root           243 Apr 15 12:53 elasticsearch.keystore
-rw-r--r-- 1 root          root          2.6K Apr 15 12:58 elasticsearch.keystore.tmp
-rw-r--r-- 1 elasticsearch elasticsearch  963 Dec 11 17:25 elasticsearch.yml
-rw-rw---- 1 elasticsearch root          2.3K Feb 29 00:13 jvm.options
-rw-rw---- 1 elasticsearch root          7.7K Feb 29 00:18 log4j2.properties
-rw-rw---- 1 elasticsearch root           473 Feb 29 00:18 role_mapping.yml
-rw-rw---- 1 elasticsearch root           197 Feb 29 00:18 roles.yml
-rw-rw---- 1 elasticsearch root             0 Feb 29 00:18 users
-rw-rw---- 1 elasticsearch root             0 Feb 29 00:18 users_roles

What do I need to do so that adding data to the keystore is possible?

Hi @slhck
Did you resolve somehow this issue?

Regards
Patryk

No solution so far. I think the fundamental issue is that the config should not live in the same directory with system generated files...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.