# Init container fails **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [August 4, 2019, 2:20pm UTC](https://discuss.elastic.co/t/init-container-fails/193684 "2019-08-04T14:20:31Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![ransoor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ransoor/32/51129_2.png) [@ransoor](https://discuss.elastic.co/u/ransoor)\ **Post date:** [August 4, 2019, 2:20pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/1 "2019-08-04T14:20:31Z")
I followed the quickstart guide version 0.9. I applied the elastic operator CRD. I deployed the elastic cluster with one node, but its status is Init:CrashLoopBackOff. After some debugging, I found that the init container elastic-internal-init-filesystem fails, specifically in the script prepare-fs.sh: "chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch chown: changing ownership of '/usr/share/elasticsearch/data': Operation not permitted" The chown command is being run by root. Am I missing anything? ---
**Author:** ![ransoor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ransoor/32/51129_2.png) [@ransoor](https://discuss.elastic.co/u/ransoor)\ **Post date:** [August 5, 2019, 7:55am UTC](https://discuss.elastic.co/t/init-container-fails/193684/2 "2019-08-05T07:55:56Z")
Ok, so I found out that the issue is happening because I have a default NFS storage class, so I created a hostPath pv instead. Now I see that the new pv is indeed bounded, but another default nfs storage class is created, and the init container elastic-internal-init-filesystem still uses it, so im having the same issue. Is this a bug? ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [August 5, 2019, 6:05pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/3 "2019-08-05T18:05:56Z")
Could you post the Elasticsearch resource you are using now here? Are you specifying a custom persistent volume claim template as described here? [https://www.elastic.co/guide/en/cloud-on-k8s/0.9/k8s-volume-claim-templates.html](https://www.elastic.co/guide/en/cloud-on-k8s/0.9/k8s-volume-claim-templates.html) Also, can you maybe share a bit more information about your Kubernetes environment? Which flavour of Kubernetes are you running and in which version? ---
**Author:** ![ransoor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ransoor/32/51129_2.png) [@ransoor](https://discuss.elastic.co/u/ransoor)\ **Post date:** [August 6, 2019, 9:04am UTC](https://discuss.elastic.co/t/init-container-fails/193684/4 "2019-08-06T09:04:18Z")
It works now. In the link you sent, it says " The name in the template must be `elasticsearch-data`" As I said, I followed the quickstart guide, and there was no mention there about the name restriction 🙂 [https://www.elastic.co/guide/en/cloud-on-k8s/0.9/k8s-quickstart.html](https://www.elastic.co/guide/en/cloud-on-k8s/0.9/k8s-quickstart.html) ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [August 7, 2019, 12:50pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/5 "2019-08-07T12:50:25Z")
We are fixing the quickstart guide to be more explicit [https://github.com/elastic/cloud-on-k8s/pull/1489](https://github.com/elastic/cloud-on-k8s/pull/1489) ---
**Author:** ![Andrii\_Litvinov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrii_litvinov/32/45475_2.png) [@Andrii\_Litvinov](https://discuss.elastic.co/u/Andrii_Litvinov)\ **Post date:** [December 20, 2019, 4:52pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/6 "2019-12-20T16:52:07Z")
I have same issue with container in Init:CrashLoopBackOff state. I use local k8s cluster deployed by Racnher and I use [https://github.com/rancher/local-path-provisioner](https://github.com/rancher/local-path-provisioner) for volumes. I tried to configure PVC and emptyDir as described here [https://www.elastic.co/guide/en/cloud-on-k8s/1.0-beta/k8s-volume-claim-templates.html](https://www.elastic.co/guide/en/cloud-on-k8s/1.0-beta/k8s-volume-claim-templates.html) and neither worked. How can I troubleshoot it further? I don't see any error massages anywhere in kubectl. ---
**Author:** ![Thibault\_Richard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibault_richard/32/50513_2.png) [@Thibault\_Richard](https://discuss.elastic.co/u/Thibault_Richard)\ **Post date:** [December 20, 2019, 5:08pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/7 "2019-12-20T17:08:40Z")
Hi Andrii, You can retrieve the logs of the init container using this command: ``` kubectl logs -c ``` Example: ``` # Retrieve the name of the init container > k get pod elasticsearch-sample-es-default-0 -o json | jq '.spec.initContainers[].name' "elastic-internal-init-filesystem" # Get the last 2 lines logged by the init container > k logs elasticsearch-sample-es-default-0 -c elastic-internal-init-filesystem | tail -2 Init script successful Script duration: 1 sec. ``` ---
**Author:** ![Andrii\_Litvinov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrii_litvinov/32/45475_2.png) [@Andrii\_Litvinov](https://discuss.elastic.co/u/Andrii_Litvinov)\ **Post date:** [December 23, 2019, 2:24pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/8 "2019-12-23T14:24:28Z")
Hi Richard, thank you for quick a response, somehow I didn't get notification about the reply. I didn't know about -c option before, and when I did that I saw following message, which is weird: > standard\_init\_linux.go:190: exec user process caused "exec format error" It appeared for filebeat as well and only on one of the nodes in my k8s cluster. I have no idea what was wrong with that node as all nodes were created by vagrant in hyper-v. I have removed and recreated the entire VM and the issues has gone, but that is still very weird and I am not sure how I would approach it if I get same error in our production cluster. ---
**Author:** ![xyphr](https://avatars.discourse-cdn.com/v4/letter/x/439d5e/32.png) [@xyphr](https://discuss.elastic.co/u/xyphr)\ **Post date:** [December 25, 2019, 3:39pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/9 "2019-12-25T15:39:25Z")
Hi, just to share my experience, this particular error `standard_init_linux.go:190: exec user process caused "exec format error` also came up for me. The reason was I had edited a file inside Elasticsearch image (which should be having LF line endings) in Windows which caused a few lines to have CRLF line endings. Also it only came up in a few pods, not all (which seems to be due to the `imagePullPolicy` being set to `IfNotPresent`). It went away when I re-normalized line endings to LF, and force pulled the fixed image. ---
**Author:** ![maxisam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxisam/32/70424_2.png) [@maxisam](https://discuss.elastic.co/u/maxisam)\ **Post date:** [June 15, 2020, 9:33pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/10 "2020-06-15T21:33:41Z")
I have similar issue. I use [https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client) as my storageclass The speed is not the issue to me because they are all VM The problem is I have the same permission error. I actually can see it creates a new folder in my nfs folder like elk-logging-elasticsearch-data-elk-logging-es-masters-0-pvc-b0e84c3f-7f8d-4891-a966-4a2de08e041a However, it is empty and I can see the error from the log chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch chown: changing ownership of '/usr/share/elasticsearch/data': Operation not permitted failed to change ownership of '/usr/share/elasticsearch/data' from 65534:65534 to elasticsearch:elasticsearch I don't understand why it can't chown the folder. I spent the whole day trying to find out a solution but fail. Hope you guys can help me. Btw, if I use local volume type, it works. I don't understand what's the problem. ---
**Author:** ![test\_1a](https://avatars.discourse-cdn.com/v4/letter/t/59ef9b/32.png) [@test\_1a](https://discuss.elastic.co/u/test_1a)\ **Post date:** [September 3, 2020, 8:41am UTC](https://discuss.elastic.co/t/init-container-fails/193684/11 "2020-09-03T08:41:46Z")
Hi, I am getting same issue. Could you help me how you resolve that? Thanks! ---
**Author:** ![Larswa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larswa/32/24798_2.png) [@Larswa](https://discuss.elastic.co/u/Larswa)\ **Post date:** [September 16, 2020, 3:49pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/12 "2020-09-16T15:49:52Z")
Seeing the same thing with the same NFS-Client storage provider set as default. How did you dig out the logs? I can only get logs from the NFS-Client pod telling me that the pvc was successfully created. ---
**Author:** ![Jorge\_Perez\_Higuera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jorge_perez_higuera/32/76164_2.png) [@Jorge\_Perez\_Higuera](https://discuss.elastic.co/u/Jorge_Perez_Higuera)\ **Post date:** [September 24, 2020, 4:41pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/13 "2020-09-24T16:41:36Z")
same problem here with nfs storage class kubectl logs quickstart-es-default-0 -c elastic-internal-init-filesystem ``` '/usr/share/elasticsearch/bin/x-pack-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-env' '/usr/share/elasticsearch/bin/x-pack-security-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-security-env' '/usr/share/elasticsearch/bin/x-pack-watcher-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-watcher-env' Files copy duration: 0 sec. chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch chown: changing ownership of '/usr/share/elasticsearch/data': Operation not permitted failed to change ownership of '/usr/share/elasticsearch/data' from 469779:469779 to elasticsearch:elasticsearch ``` ---
**Author:** ![Emil\_Vanneback](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emil_vanneback/32/84542_2.png) [@Emil\_Vanneback](https://discuss.elastic.co/u/Emil_Vanneback)\ **Post date:** [February 25, 2021, 1:19pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/14 "2021-02-25T13:19:09Z")
I have the same error as above when using nfs as storageclass. ```auto chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch chown: changing ownership of '/usr/share/elasticsearch/data': Operation not permitted failed to change ownership of '/usr/share/elasticsearch/data' from 65534:65534 to elasticsearch:elasticsearch ``` Is there any update / progress on this? ---
**Author:** ![likarum](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/likarum/32/87249_2.png) [@likarum](https://discuss.elastic.co/u/likarum)\ **Post date:** [April 17, 2021, 10:30pm UTC](https://discuss.elastic.co/t/init-container-fails/193684/15 "2021-04-17T22:30:44Z")
I think you have an issue on your nfs server. Check if root\_squash is disable on your nfs server configuration. > **[Unix security | Root squash](https://en.wikipedia.org/wiki/Unix_security#Root_squash)** > > Root squash is a special mapping of the remote superuser (root) identity when using identity authentication (local user is the same as remote user). Under root squash, a client's uid 0 (root) is mapped to 65534 (nobody). It is primarily a feature of NFS but may be available on other systems as well. > Root squash is a technique to avoid privilege escalation on the client machine via suid executables Setuid. Without root squash, an attacker can generate suid binaries on the server that are executed... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:19am UTC](https://discuss.elastic.co/t/init-container-fails/193684/16 "2022-11-04T07:19:53Z")