"Device or resource busy" when adding to keystore in Docker Compose

Elastic Stack Elasticsearch
1

I want to persist the elasticsearch.keystore file in a Docker Compose project.

Obviously there is a Catch-22 in that I could mount an existing keystore file, but I don't have one yet if I start with a new setup.

So first, I run this Compose-file via docker-compose -f docker-compose.setup.yml up:

version: '3.7'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: 7.6.2
    volumes:
      - type: bind
        source: ./elasticsearch/config/
        target: /config
      - type: bind
        source: ./elasticsearch/container_setup.sh
        target: /usr/share/elasticsearch/container_setup.sh
    command: ./container_setup.sh
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      ELASTIC_PASSWORD: ${ELASTICSEARCH_ELASTIC_PASSWORD:-changeme}

The container_setup.sh file is as follows:

#!//usr/bin/env bash

echo "=== CREATE Keystore ==="
echo "Elastic password is: $ELASTIC_PASSWORD"

if [ -f /config/elasticsearch/elasticsearch.keystore ]; then
    echo "Remove old elasticsearch.keystore from host config"
    rm /config/elasticsearch/elasticsearch.keystore
fi

echo "Creating keystore"
if [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]]; then
    echo "Keystore already exists"
else
    echo "Creating new keystore"
    /usr/share/elasticsearch/bin/elasticsearch-keystore create
fi

echo "Setting bootstrap.password..."
echo "$ELASTIC_PASSWORD" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x 'bootstrap.password'

echo "Moving keystore"
mv /usr/share/elasticsearch/config/elasticsearch.keystore /config/elasticsearch.keystore

What this does is create a new elasticsearch.keystore file in config on my host. Interestingly it will be owned by root:root:

➜ ll elasticsearch/config 
total 8
-rw-rw---- 1 root   root   243 Apr 15 14:53 elasticsearch.keystore
-rw-r--r-- 1 werner werner 963 Dec 11 18:25 elasticsearch.yml

Now I start the "real" Elastic service with the following Compose file:

version: '3.7'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: 7.6.2
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
      - type: bind
        source: ./elasticsearch/config/elasticsearch.keystore
        target: /usr/share/elasticsearch/config/elasticsearch.keystore
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      ELASTIC_PASSWORD: ${ELASTICSEARCH_ELASTIC_PASSWORD:-changeme}
    restart: unless-stopped

volumes:
    elasticsearch: {}

This works fine. The real issue occurs when I now try to add an entry to the keystore. So I run:

➜ docker-compose run --rm elasticsearch /bin/bash

And I try to add the file:

bin/elasticsearch-keystore add-file "gcs.client.foo.credentials_file" /tmp/auth.json

What I get is the following:

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/e
lasticsearch.keystore: Device or resource busy
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
        at java.base/java.nio.file.Files.move(Files.java:1425)
        at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:523)
        at org.elasticsearch.common.settings.AddFileKeyStoreCommand.execute(AddFileKeyStoreCommand.java:94)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)

The permissions in the container config directory are as follows before running that command:

[root@44dff2478ddc elasticsearch]# ls -lha config/
total 44K
drwxrwxr-x 1 elasticsearch root          4.0K Apr 15 12:58 .
drwxrwxr-x 1 elasticsearch root          4.0K Feb 29 00:20 ..
-rw-rw---- 1 root          root           243 Apr 15 12:53 elasticsearch.keystore
-rw-r--r-- 1 root          root          2.6K Apr 15 12:58 elasticsearch.keystore.tmp
-rw-r--r-- 1 elasticsearch elasticsearch  963 Dec 11 17:25 elasticsearch.yml
-rw-rw---- 1 elasticsearch root          2.3K Feb 29 00:13 jvm.options
-rw-rw---- 1 elasticsearch root          7.7K Feb 29 00:18 log4j2.properties
-rw-rw---- 1 elasticsearch root           473 Feb 29 00:18 role_mapping.yml
-rw-rw---- 1 elasticsearch root           197 Feb 29 00:18 roles.yml
-rw-rw---- 1 elasticsearch root             0 Feb 29 00:18 users
-rw-rw---- 1 elasticsearch root             0 Feb 29 00:18 users_roles

What do I need to do so that adding data to the keystore is possible?

2

Hi @slhck
Did you resolve somehow this issue?

Regards
Patryk

3

No solution so far. I think the fundamental issue is that the config should not live in the same directory with system generated files...

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.