Deploy an Elasticsearch cluster with QuickStart

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I have deployed ECK in a cluster and the quickstart has stayed in the "ApplyingChanges" phase and the quickstart pods are like this:

NAMESPACE NAME HEALTH NODES VERSION PHASE AGE
monitoring quickstart unknown 8.3.1 ApplyingChanges 98m

NAMESPACE NAME READY STATUS RESTARTS AGE
monitoring quickstart-es-default-0 0/1 Init:0/3 0 26s
monitoring quickstart-es-default-1 0/1 Init:0/3 0 26s
monitoring quickstart-es-default-2 0/1 Init:0/3 0 28s

2

Init:0/3: only 2 init containers are created by default: elastic-internal-init-filesystem and elastic-internal-suspend. Could you check why is there a third one and check if there're any logs from it?

3

Good morning Michel, first of all thank you for the answer. I have three pods deployed from quickstart, so it looks like one on each worker, doing a describe on each shows the following: Normal Scheduled 8s default-scheduler Successfully assigned monitoring/quickstart-es-default-2 to worker3 Normal Scheduled 32s default-scheduler Successfully assigned monitoring/quickstart-es-default-0 to worker1 Normal Scheduled 7s default-scheduler Successfully assigned monitoring/quickstart-es-default-1 to worker2

4

My question was more about the number of init containers than the number of Pods. Could you run the following command please?

kubectl get pod/quickstart-es-default-0 -o jsonpath='{ .spec.initContainers[*].command }'
5

This is the result @michael.morello
["bash","-c","/mnt/elastic-internal/scripts/prepare-fs.sh"] ["bash","-c","/mnt/elastic-internal/scripts/suspend.sh"] ["sh","-c","sysctl -w vm.max_map_count=262144"]

6

This is not exactly the quickstart :smile:

Could you run this command:

kubectl get pod/quickstart-es-default-0 -o jsonpath='{ .status }'

7

{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2022-07-07T08:49:22Z","message":"containers with incomplete status: [elastic-internal-init-filesystem elastic-internal-suspend sysctl]","reason":"ContainersNotInitialized","status":"False","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2022-07-07T08:49:22Z","message":"containers with unready status: [elasticsearch]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":"2022-07-07T08:49:22Z","message":"containers with unready status: [elasticsearch]","reason":"ContainersNotReady","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2022-07-07T08:49:22Z","status":"True","type":"PodScheduled"}],"containerStatuses":[{"image":"docker.elastic.co/elasticsearch/elasticsearch:8.3.1","imageID":"","lastState":{},"name":"elasticsearch","ready":false,"restartCount":0,"started":false,"state":{"waiting":{"reason":"PodInitializing"}}}],"hostIP":"xxxxxxx","initContainerStatuses":[{"image":"docker.elastic.co/elasticsearch/elasticsearch:8.3.1","imageID":"","lastState":{},"name":"elastic-internal-init-filesystem","ready":false,"restartCount":0,"state":{"waiting":{"reason":"PodInitializing"}}},{"image":"docker.elastic.co/elasticsearch/elasticsearch:8.3.1","imageID":"","lastState":{},"name":"elastic-internal-suspend","ready":false,"restartCount":0,"state":{"waiting":{"reason":"PodInitializing"}}},{"image":"docker.elastic.co/elasticsearch/elasticsearch:8.3.1","imageID":"","lastState":{},"name":"sysctl","ready":false,"restartCount":0,"state":{"waiting":{"reason":"PodInitializing"}}}],"phase":"Pending","qosClass":"Burstable","startTime":"2022-07-07T08:49:22Z"}

8

Phase is still Pending, also the Pod does not seem to have a podIP, this may prevent the operator from generating the certificates, and consequently also prevent the Pod from starting.

Could you double check the podIP by running kubectl get pod/quickstart-es-default-0 -o wide

Could you also:

  • Get the containers logs (if any): kubectl logs --all-containers pod/quickstart-es-default-0
  • Check if there is any error in the operator logs
  • Check if there is anything wrong in the K8S events.
9

kubectl logs --all-containers pod/quickstart-es-default-0 -n monitoring
Error from server (BadRequest): container "sysctl" in pod "quickstart-es-default-0" is waiting to start: PodInitializing

10

operator's son records

"log.level":"info","@timestamp":"2022-07-07T11:16:43.080Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"2.2.0+02f250eb","service.type":"eck","ecs.version":"1.4.0","namespace":"monitoring","pod_name":"quickstart-es-default-2"}
{"log.level":"info","@timestamp":"2022-07-07T11:16:43.084Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"2.2.0+02f250eb","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"monitoring","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2022-07-07T11:16:43.096Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.2.0+02f250eb","service.type":"eck","ecs.version":"1.4.0","namespace":"monitoring","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2022-07-07T11:16:43.096Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.2.0+02f250eb","service.type":"eck","ecs.version":"1.4.0","iteration":191787,"namespace":"monitoring","es_name":"quickstart","took":0.316895309}
11

@michael.morello FailedMount pod/quickstart-es-default-2 Unable to attach or mount volumes: unmounted volumes=[elastic-internal-transport-certificates elastic-internal-elasticsearch-bin-local elastic-internal-probe-user elastic-internal-elasticsearch-config elastic-internal-http-certificates elasticsearch-logs elastic-internal-scripts elastic-internal-elasticsearch-plugins-local elastic-internal-unicast-hosts elastic-internal-remote-certificate-authorities elastic-internal-elasticsearch-config-local elastic-internal-xpack-file-realm elasticsearch-data downward-api], unattached volumes=[elastic-internal-transport-certificates elastic-internal-elasticsearch-bin-local elastic-internal-probe-user elastic-internal-elasticsearch-config elastic-internal-http-certificates elasticsearch-logs elastic-internal-scripts elastic-internal-elasticsearch-plugins-local elastic-internal-unicast-hosts elastic-internal-remote-certificate-authorities elastic-internal-elasticsearch-config-local elastic-internal-xpack-file-realm elasticsearch-data downward-api]: timed out waiting for the condition

13
  • Check if there is anything wrong in the K8S events.
14

that trace is from the events : kubectl get events

15

Hello @michael.morello any ideas about the error?

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.