AlreadyClosedException error for a Elasticsearch image from `docker commit`

Elastic Stack Elasticsearch
1

We have a use case to preload some data (e.g. creating some indexes ) into a running Elasticsearch container and once finished, docker commit it to create a data-preloaded image that can be used later.

Everything works on docker.elastic.co/elasticsearch/elasticsearch:7.17.16 but things are not working on the latest image 7.17.27.

After some debugging and narrowing down, not work even without data loading (below codes).

# Script to create a image from `docker commit`
#!/usr/bin/env bash
BUILD_NUMBER="1"
REPOSITORY_NAME="nerd-repo"
REGISTRY="102642240800.dkr.ecr.ap-southeast-2.amazonaws.com"

echo "Starting docker compose..."
docker compose up -d

echo "waiting Elasticsearch cluster to settledown..."
sleep 10

echo "Stop the container before committing..."
docker stop elastic_nerapi

echo "Commit docker container..."
docker commit elastic_nerapi $REGISTRY/$REPOSITORY_NAME:$BUILD_NUMBER
docker tag $REGISTRY/$REPOSITORY_NAME:$BUILD_NUMBER $REGISTRY/$REPOSITORY_NAME:latest

docker compose down

docker run \
  --name nerd_sidecar \
  -p 9200:9200 \
  -p 9300:9300 \
  -e cluster.name=nerd-cluster \
  -e discovery.type=single-node \
  -e ES_JAVA_OPTS='-Xms3g -Xmx3g' \
  102642240800.dkr.ecr.ap-southeast-2.amazonaws.com/nerd-repo:$BUILD_NUMBER

trap : 0

// docker-compose.yml
version: "2.2"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.27
    container_name: elastic_nerapi
    environment:
      - cluster.name=nerd-cluster
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms3g -Xmx3g"

    ports:
      - 9200:9200
    networks:
      - esnet

networks:
  esnet:

// Error message
{"type": "server", "timestamp": "2025-02-17T21:44:43,657Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "nerd-cluster", "node.name": "f5b2da2ba459", "message": "heap size [3gb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2025-02-17T21:44:43,658Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "nerd-cluster", "node.name": "f5b2da2ba459", "message": "uncaught exception in thread [main]", 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2025-02-17T21:44:43.653676Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2025-02-17T21:44:39.694362012Z))",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.27.jar:7.17.27]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.27.jar:7.17.27]",
"Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2025-02-17T21:44:43.653676Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2025-02-17T21:44:39.694362012Z))",
"at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) ~[lucene-core-8.11.3.jar:8.11.3 baa7c80af4278cc8951a344d8e9320386588d12d - houstonputman - 2024-02-05 15:02:58]",
"at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1105) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:867) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1280) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:342) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.27.jar:7.17.27]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.27.jar:7.17.27]",
"... 6 more"] }
uncaught exception in thread [main]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2025-02-17T21:44:43.653676Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2025-02-17T21:44:39.694362012Z))
        at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191)
        at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1105)
        at org.elasticsearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:867)
        at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1280)
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:342)
        at org.elasticsearch.node.Node.<init>(Node.java:429)
        at org.elasticsearch.node.Node.<init>(Node.java:309)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
        at org.elasticsearch.cli.Command.main(Command.java:77)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/nerd-cluster.log

any input is appreciated.

2

Hello, this method is not supported. See Snapshot and restore | Elasticsearch Guide [8.17] | Elastic which states:

Taking a snapshot is the only reliable and supported way to back up a cluster. You cannot back up an Elasticsearch cluster by making copies of the data directories of its nodes. There are no supported methods to restore any data from a filesystem-level backup. If you try to restore a cluster from such a backup, it may fail with reports of corruption or missing files or other data inconsistencies, or it may appear to have succeeded having silently lost some of your data.

A copy of the data directories of a cluster’s nodes does not work as a backup because it is not a consistent representation of their contents at a single point in time. You cannot fix this by shutting down nodes while making the copies, nor by taking atomic filesystem-level snapshots, because Elasticsearch has consistency requirements that span the whole cluster. You must use the built-in snapshot functionality for cluster backups.