# ECK 1.0.0-beta1 during node startup pod goes to CrashLoopBackOff **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [December 9, 2019, 4:12pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154 "2019-12-09T16:12:49Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![sfgroups1](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sfgroups1](https://discuss.elastic.co/u/sfgroups1)\ **Post date:** [December 9, 2019, 4:12pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/1 "2019-12-09T16:12:49Z")
Hi, i have 6 node ECK cluster, when ever node reboots elastic 7.5 pod is not starting it goes in 'crashloop node'. If I delete the pod, then its starting properly. I am un shure why POD goes into crashloop. ``` # kgpw -w NAME READY STATUS RESTARTS AGE elastic-operator-0 1/1 Running 8 2d22h elk-prd-es-default-0 1/1 Running 0 22h elk-prd-es-default-1 1/1 Running 0 2d16h elk-prd-es-default-2 1/1 Running 0 2d16h elk-prd-es-default-3 1/1 Running 0 17s elk-prd-es-default-4 0/1 Init:CrashLoopBackOff 9 22h elk-prd-es-default-4 0/1 Init:1/3 10 22h elk-prd-es-default-4 0/1 Init:Error 10 22h elk-prd-es-default-4 0/1 Init:CrashLoopBackOff 10 22h ``` Here is the events: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 21m (x5767 over 21h) kubelet, ecknode04 Back-off restarting failed container Normal SandboxChanged 17m kubelet, ecknode04 Pod sandbox changed, it will be killed and re-created. Normal Pulled 17m kubelet, ecknode04 Container image "docker.elastic.co/elasticsearch/elasticsearch:7.5.0" already present on machine Normal Created 17m kubelet, ecknode04 Created container elastic-internal-init-filesystem Normal Started 17m kubelet, ecknode04 Started container elastic-internal-init-filesystem Normal Pulled 17m (x4 over 17m) kubelet, ecknode04 Container image "docker.elastic.co/elasticsearch/elasticsearch:7.5.0" already present on machine Normal Created 17m (x4 over 17m) kubelet, ecknode04 Created container elastic-internal-init-keystore Normal Started 17m (x4 over 17m) kubelet, ecknode04 Started container elastic-internal-init-keystore Warning BackOff 2m50s (x77 over 17m) kubelet, ecknode04 Back-off restarting failed container ``` any help to resolve this issue? ---
**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\ **Post date:** [December 9, 2019, 5:01pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/2 "2019-12-09T17:01:03Z")
Hi @sfgroups1, if you look at the logs of the pod (docs here: [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-troubleshooting.html#k8s-get-elasticsearch-logs](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-troubleshooting.html#k8s-get-elasticsearch-logs)) you may get a better idea of why it is crashing. ---
**Author:** ![sfgroups1](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sfgroups1](https://discuss.elastic.co/u/sfgroups1)\ **Post date:** [December 9, 2019, 10:09pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/3 "2019-12-09T22:09:08Z")
log message not showing any error. it says pod started. something else causing pod not get to ready status. ``` kgp |grep elk-prd-es-default-4 elk-prd-es-default-4 0/1 Init:CrashLoopBackOff 80 28h k logs elk-prd-es-default-4 |tail -1 {"type": "server", "timestamp": "2019-12-08T17:03:29,608Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-prd", "node.name": "elk-prd-es-default-4", "message": "started", "cluster.uuid": "3d9MXxV2S4-226M5VR7cjA", "node.id": "LEnpjkmaSbm6ROM2XQIDSg" } ``` node log show this error message. `Error syncing pod 16eff8e8-6b1a-4de7-b031-6af8d78ddb12 ("elk-prd-es-default-4_elastic-system(16eff8e8-6b1a-4de7-b031-6af8d78ddb12)"), skipping: failed to "StartContainer" for "elastic-internal-init-keystore" with CrashLoopBackOff: "back-off 5m0s restarting failed container=elastic-internal-init-keystore pod=elk-prd-es-default-4_elastic-system(16eff8e8-6b1a-4de7-b031-6af8d78ddb12)"` ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [December 10, 2019, 10:15am UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/4 "2019-12-10T10:15:00Z")
Can you post your Elasticsearch yaml manifest? Also can you give us the output of the init container logs? ```auto kubectl logs elk-prd-es-default-4 -c elastic-internal-init-keystore ``` ---
**Author:** ![sfgroups1](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sfgroups1](https://discuss.elastic.co/u/sfgroups1)\ **Post date:** [December 10, 2019, 1:46pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/5 "2019-12-10T13:46:57Z")
> [@sebgl](#): > > kubectl logs elk-prd-es-default-4 -c elastic-internal-init-keystore Here is the output: unsure why its looking for terminal. ``` # kubectl logs elk-prd-es-default-0 -c elastic-internal-init-keystore + echo 'Initializing keystore.' + /usr/share/elasticsearch/bin/elasticsearch-keystore create Initializing keystore. Exception in thread "main" java.lang.IllegalStateException: unable to read from standard input; is standard input open and a tty attached? at org.elasticsearch.cli.Terminal$SystemTerminal.readText(Terminal.java:207) at org.elasticsearch.cli.Terminal.promptYesNo(Terminal.java:140) at org.elasticsearch.common.settings.CreateKeyStoreCommand.execute(CreateKeyStoreCommand.java:43) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41) ``` ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [December 10, 2019, 2:01pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/6 "2019-12-10T14:01:58Z")
This looks like a wrong keystore init container command. Can you please post your elasticsearch resource yaml manifest? ---
**Author:** ![sfgroups1](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sfgroups1](https://discuss.elastic.co/u/sfgroups1)\ **Post date:** [December 10, 2019, 7:53pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/7 "2019-12-10T19:53:06Z")
Here is the yaml file. I have this issue only during server restart, if I delete the crashloop pod, then pod starting properly. ``` apiVersion: elasticsearch.k8s.elastic.co/v1beta1 kind: Elasticsearch metadata: name: elk-prd spec: version: 7.5.0 nodeSets: - name: default count: 5 config: node.master: true node.data: true node.ingest: true node.store.allow_mmap: false podDisruptionBudget: spec: maxUnavailable: 2 minAvailable: 3 selector: matchLabels: elasticsearch.k8s.elastic.co/cluster-name: elk-prd volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 490Gi storageClassName: local-storage ``` ---
**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\ **Post date:** [July 16, 2020, 6:31am UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/8 "2020-07-16T06:31:29Z")
Where are we with this issue? I am seeing the exact same one in my ECK Cluster v1.1.2. We run a kubernetes cluster at AWS that is being suspended every night to safe money. Before adding the `secureSettings` section the elasticsearch.yaml was working fine. The cluster was coming up every morning without any issue. However, this morning the cluster stuck in crashloopbackoff. We started with the usage of the `secureSettings` yesterday to add some gcs-credentials to the cluster. This lead unfortunately to the below listed issue: Snipped of the elasticsearch.yaml ```auto apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: elastic spec: version: {{ $.Chart.AppVersion }} secureSettings: - secretName: elastic-es-gcs-credentials nodeSets: - name: master count: 3 config: node.master: true node.data: false node.ingest: false ``` Describe of the failing pod ```auto elastic-internal-init-keystore: Container ID: docker://7eb7d89d6150d2a5983a9ea9461e7cea9c541a1443c181e1dbf96fa7f96df867 Image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0 Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:161bc8c7054c622b057324618a2e8bc49ae703e64901b141a16d9c8bdd3b82f9 Port: Host Port: Command: /usr/bin/env bash -c #!/usr/bin/env bash set -eux echo "Initializing keystore." # create a keystore in the default data path /usr/share/elasticsearch/bin/elasticsearch-keystore create # add all existing secret entries into it for filename in /mnt/elastic-internal/secure-settings/*; do [[-e "$filename"]] || continue # glob does not match key=$(basename "$filename") echo "Adding "$key" to the keystore." /usr/share/elasticsearch/bin/elasticsearch-keystore add-file "$key" "$filename" done echo "Keystore initialization successful." State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Thu, 16 Jul 2020 07:31:32 +0200 Finished: Thu, 16 Jul 2020 07:31:33 +0200 Ready: False Restart Count: 14 ``` Logs of the failing init container show the following: ```auto [raulgs@raulgs-xm1 elastic]$ klogs -f pod/elastic-es-data-0 elastic-internal-init-keystore + echo 'Initializing keystore.' + /usr/share/elasticsearch/bin/elasticsearch-keystore create Initializing keystore. Exception in thread "main" java.lang.IllegalStateException: unable to read from standard input; is standard input open and a tty attached? at org.elasticsearch.cli.Terminal$SystemTerminal.readText(Terminal.java:273) at org.elasticsearch.cli.Terminal.promptYesNo(Terminal.java:152) at org.elasticsearch.common.settings.CreateKeyStoreCommand.execute(CreateKeyStoreCommand.java:51) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43) ``` Deleting the pod so that it gets re-deployed by k8s fixes the issue. However, this is only a workaround. After pod deletion and re-deployment the log of the keystore init container looks like that: ```auto [raulgs@raulgs-xm1 elastic]$ klogs -f pod/elastic-es-data-0 elastic-internal-init-keystore + echo 'Initializing keystore.' + /usr/share/elasticsearch/bin/elasticsearch-keystore create Initializing keystore. Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore + for filename in '/mnt/elastic-internal/secure-settings/*' + [[-e /mnt/elastic-internal/secure-settings/gcs.client.default.credentials_file]] ++ basename /mnt/elastic-internal/secure-settings/gcs.client.default.credentials_file + key=gcs.client.default.credentials_file + echo 'Adding gcs.client.default.credentials_file to the keystore.' + /usr/share/elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /mnt/elastic-internal/secure-settings/gcs.client.default.credentials_file Adding gcs.client.default.credentials_file to the keystore. + echo 'Keystore initialization successful.' Keystore initialization successful. ``` ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [July 16, 2020, 7:25am UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/9 "2020-07-16T07:25:55Z")
> [@raulgs](#): > > We run a kubernetes cluster at AWS that is being suspended every night to safe money. I suspect this is the underlying cause for the issue you are seeing. When you suspend and resume the cluster in the morning the init containers for the existing Pods are run again. The init-keystore init container script is not expecting repeated runs and the `elasticsearch-keystore` command will ask for user permission if it encounters an existing keystore in the config directory which causes the error you are seeing. This is a known issue see [Init containers should tolerate restarts · Issue #3294 · elastic/cloud-on-k8s · GitHub](https://github.com/elastic/cloud-on-k8s/issues/3294) and a fix will be included in ECK 1.2 ---
**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\ **Post date:** [July 16, 2020, 7:28am UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/10 "2020-07-16T07:28:23Z")
Yeah that looks pretty much like it. Do you know when we can expect it to be released? Or is there a workaround that I could use in the meantime? ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [July 21, 2020, 3:15pm UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/11 "2020-07-21T15:15:22Z")
ECK 1.2. is available as of today [https://www.elastic.co/guide/en/cloud-on-k8s/1.2/](https://www.elastic.co/guide/en/cloud-on-k8s/1.2/) ---
**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\ **Post date:** [July 22, 2020, 5:54am UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/12 "2020-07-22T05:54:36Z")
I already saw it yesterday and upgrade directly. I can confirm the issue with re-initialization of the key store is fixed. Thanks a lot guys. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:36am UTC](https://discuss.elastic.co/t/eck-1-0-0-beta1-during-node-startup-pod-goes-to-crashloopbackoff/211154/13 "2022-11-04T07:36:41Z")