Elasticsearch will not start in Kubernetes if I mount a persistent volume to /usr/share/elasticsearch/data

Elastic Stack Elasticsearch
1

Hello, all,
I have a very strange problem that I've been testing and working on for a couple of days. I am running elasticsearch in Kubernetes, trying to get the first pod up and running. I have it running fine, it starts, generates the auto config for security (creates the config/certs directory, modifies the elasticsearch.yml with the xpack.security settings, etc) and it starts up and runs just fine.

Now, I need to have persistent storage for the /data directory, so I create a new persistent volume, volume mount it to /usr/share/elasticsearch/data, run an initContainer to set the permissions and ownership to elasticsearch:elasticsearch user/group, all of that works great. But when this persistent volume is mounted, elasticsearch fails to start. It fails bootstrap checks because the "certs" directory does not get created and the elasticsearch.yml does not get updated. I remove the persistent volume mount and let it default back to the /data directory embedded in the image itself, and elasticsearch startup up and everything works as it should.

Some files do get created in the /data directory (_state, snapshot_cache,nodes, etc) before elasticsearch errors out, so I don't believe this is a permissions issue on the persistent datastore.

Another clue, an "indices" sub directory gets created in the /data directory when I don't use the Persistent Volume, it does not get created when the persistent volume is mounted. I don't know if that is the root cause, or if that indices just doesn't get created because elasticsearch crashes and exits before it gets that far.

I went into the log4j2.properties and turned everything I could find into "debug" but still no useful clues in the elasticsearch startup output, even with all of the debug output.

Has anyone run into this before, and can anyone point me in the right direction? I'm completely stumped as to what is actually preventing the security auto config from finishing just because there is a persistent volume present.

2

Below is the log excerpt ERROR when I have a persistent volume mounted. These bootstrap checks fail because the elasticsearch.yml did not get updated with cluster.initial_master_nodes and xpack.security items. When I remove the persistent volume, the YML does get updated, and elasticsearch starts up just fine.

{"@timestamp":"2025-07-10T20:07:47.555Z", "log.level":"ERROR", "message":"node validation exception\n[2] bootstrap checks failed. You must address the points described in the following [2] lines before starting Elasticsearch. For more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.15/bootstrap-checks.html]\nbootstrap check failure [1] of [2]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured; for more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.15/_discovery_configuration_check.html]\nbootstrap check failure [2] of [2]: Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]; for more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.15/bootstrap-checks-xpack.html#bootstrap-checks-tls]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"test-docstore-ds01","elasticsearch.cluster.name":"test-docstore-document-store"}

3

Well shortly after posting this, I found the solution...it's here:

Cases When Security Auto Config is skipped.
Elasticsearch won't run the auto config if it thinks the environment already exists, one of those criteria it uses is that the /data directory exists AND is not empty. When I mounted the persistent volume, it had a system "lost+found" empty directory in it, but that was enough to cause Elasticsearch to rule the data directory as non-empty and skip the security auto config, then when it tried to start, essential settings were missing from the elasticsearch.yml. I deleted that lost+found directory and restarted and everything auto deployed like it should have.

1 Like