Permission denied error when starting Elasticsearch as Singularity container

[lucasl (Lucas)]

Hello,

I am trying to run single node Elasticsearch instance on a HPC cluster. To do this, I am converting the Elasticsearch docker container as a singularity container. When I launch the container itself I get the following error:

$ singularity exec --overlay overlay.img elastic.sif /usr/share/elasticsearch/bin/elasticsearch
Could not create auto-configuration directory
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
output:
[0.000s][error][logging] Error opening log file 'logs/gc.log': Permission denied
[0.000s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
error:
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
        at org.elasticsearch.server.cli.JvmOption.flagsFinal(JvmOption.java:113)
        at org.elasticsearch.server.cli.JvmOption.findFinalOptions(JvmOption.java:80)
        at org.elasticsearch.server.cli.MachineDependentHeap.determineHeapSettings(MachineDependentHeap.java:59)
        at org.elasticsearch.server.cli.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:132)
        at org.elasticsearch.server.cli.JvmOptionsParser.determineJvmOptions(JvmOptionsParser.java:90)
        at org.elasticsearch.server.cli.ServerProcess.createProcess(ServerProcess.java:211)
        at org.elasticsearch.server.cli.ServerProcess.start(ServerProcess.java:106)
        at org.elasticsearch.server.cli.ServerProcess.start(ServerProcess.java:89)
        at org.elasticsearch.server.cli.ServerCli.startServer(ServerCli.java:213)
        at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:90)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
        at org.elasticsearch.cli.Command.main(Command.java:50)
        at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

If I understand correctly, Elasticsearch is trying to create a logfile in /var/log/elasticsearch but does not have the correct permissions. So I created the following recipe to create the folders and set the permission such that any process can write into the log directory. My recipe is the following:

Bootstrap: docker
From: elasticsearch:8.3.1

%files
    elasticsearch.yml /usr/share/elasticsearch/config/

%post
    mkdir -p /var/log/elasticsearch
    chown -R elasticsearch:elasticsearch /var/log/elasticsearch
    chmod -R 777 /var/log/elasticsearch

    mkdir -p /var/data/elasticsearch
    chown -R elasticsearch:elasticsearch /var/data/elasticsearch
    chmod -R 777 /var/data/elasticsearch

The elasticsearch.yml file has the following content:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

discovery.type: single-node
ingest.geoip.downloader.enabled: false

After building this recipe the directories do seem to get created correctly:

$ singularity exec elastic.sif ls -alh /var/log/ 
total 569K
drwxr-xr-x  4 root          root           162 Jul  8 14:43 .
drwxr-xr-x 12 root          root           172 Jul  8 14:43 ..
-rw-r--r--  1 root          root          7.7K Jun 29 17:29 alternatives.log
drwxr-xr-x  2 root          root            69 Jun 29 17:29 apt
-rw-r--r--  1 root          root           58K May 31 11:43 bootstrap.log
-rw-rw----  1 root          utmp             0 May 31 11:43 btmp
-rw-r--r--  1 root          root          187K Jun 29 17:30 dpkg.log
drwxrwxrwx  2 elasticsearch elasticsearch    3 Jul  8 14:43 elasticsearch
-rw-r--r--  1 root          root           32K Jun 29 17:30 faillog
-rw-rw-r--  1 root          utmp          286K Jun 29 17:30 lastlog
-rw-rw-r--  1 root          utmp             0 May 31 11:43 wtmp

But when I launch the container I get the permission denied error listed above.

What is missing here? What permissions is Elasticsearch expecting?

[lucasl (Lucas)]

The following workaround seems to be working for me now:

When launching the singularity container, the elasticsearch process is executed inside the container with the same UID as my own UID (the user that is launching the singularity container with singularity exec). The elasticsearch container is configured to run elasticsearch with the a separate user elasticsearch that exists inside the container. The issue is that singularity (unlike docker) will run every process inside the container with my own UID and not the elasticsearch UID, resulting in the error above.

To work around this, I created a base ubuntu singularity image and then installed elasticsearch into the container following these installation instructions. Because the installation was performed with my system user and UID, the entire elasticsearch installation belongs to my system user and not a separate elasticsearch user. Then I can launch the elasticsearch service inside the container.