Timeout and failure in creating ECK in kubernetes version 1.14.4

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

The same yaml can be created successfully under version 1.16.3

kubectl create -f test.yaml
Error from server (Timeout): error when creating "test.yaml": Timeout: request did not complete within requested timeout 30s

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
Check the operator log and find no errors

> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
>   name: elastic-local-volume
> provisioner: kubernetes.io/no-provisioner
> volumeBindingMode: WaitForFirstConsumer
> 
> ---
> apiVersion: v1
> kind: PersistentVolume
> metadata:
>   name: es-data-pv-0
> spec:
>   capacity:
>     storage: 5Gi
>   accessModes:
>     - ReadWriteOnce
>   persistentVolumeReclaimPolicy: Recycle
>   storageClassName: elastic-local-volume
>   local:
>     path: /data/elastic-pv0/
>   nodeAffinity:
>     required:
>       nodeSelectorTerms:
>         - matchExpressions:
>             - key: kubernetes.io/hostname
>               operator: In
>               values:
>                 - '192.168.10.184'
> ---
> apiVersion: v1
> kind: PersistentVolume
> metadata:
>   name: es-data-pv-1
> spec:
>   capacity:
>     storage: 50Gi
>   accessModes:
>     - ReadWriteOnce
>   persistentVolumeReclaimPolicy: Recycle
>   storageClassName: elastic-local-volume
>   local:
>     path: /data/elastic-pv1/
>   nodeAffinity:
>     required:
>       nodeSelectorTerms:
>         - matchExpressions:
>             - key: kubernetes.io/hostname
>               operator: In
>               values:
>                 - '192.168.10.184'


> apiVersion: elasticsearch.k8s.elastic.co/v1
> kind: Elasticsearch
> metadata:
>   name: elastic
> spec:
>   version: 7.6.2
>   image: ccr.ccs.tencentyun.com/elk-oss/elasticsearch:7.6.2
>   nodeSets:
>   - name: elastic-master
>     count: 1
>     config:
>       node.master: true
>       node.data: false
>       node.ingest: false
>       node.ml: false
>       node.store.allow_mmap: false
>       index.store.type: niofs
>       cluster.remote.connect: false
>     podTemplate:
>       spec:
>         containers:
>         - name: elasticsearch
>           resources:
>             limits:
>               memory: 2Gi
>               cpu: 2
>           env:
>           - name: ES_JAVA_OPTS
>             value: "-Xms2g -Xmx2g"
>     volumeClaimTemplates:
>     - metadata:
>         name: elasticsearch-data
>       spec:
>         accessModes:
>         - ReadWriteOnce
>         resources:
>           requests:
>             storage: 5Gi
>         storageClassName: elastic-local-volume
> 
>   - name: elastic-node
>     count: 1
>     config:
>       node.master: false
>       node.data: true
>       node.ingest: true
>       node.ml: false
>       node.store.allow_mmap: false
>       index.store.type: niofs
>       cluster.remote.connect: false
>     PodTemplate:
>       spec:
>         containers:
>         - name: elasticsearch
>           resources:
>             limits:
>               memory: 4Gi
>               cpu: 2
>           env:
>           - name: ES_JAVA_OPTS
>             value: "-Xms4g -Xmx4g"
>     volumeClaimTemplates:
>     - metadata:
>         name: elasticsearch-data
>       spec:
>         accessModes:
>         - ReadWriteOnce
>         resources:
>           requests:
>             storage: 5Gi
>         storageClassName: elastic-local-volume
2

Hi,
Maybe you should first create PV/PVC, check them and then create the elasticsearch cluster?

3

@abrx Thank you for your reply. The problem has been solved

reference: https://github.com/elastic/cloud-on-k8s/issues/1673

4

We have since updated our documentation which now contains a section to troubleshoot the validating webhook https://www.elastic.co/guide/en/cloud-on-k8s/1.1/k8s-webhook-troubleshooting.html

1 Like
5

Thanks for the tips

6

I am a little curious, why only elasticsearch will be affected by the admission webhook, and after deleting the admission webhook, the operator log will always prompt error, is there no problem?

kubectl create -f .
Error from server (Timeout): error when creating "1master-2node.yaml": Timeout: request did not complete within requested timeout 30s
Error from server (AlreadyExists): error when creating "operator-apm.yaml": apmservers.apm.k8s.elastic.co "apm-server" already exists
Error from server (AlreadyExists): error when creating "operator-kibana.yaml": kibanas.kibana.k8s.elastic.co "kibana" already exists
Error from server (AlreadyExists): error when creating "pv-3c-30g.yaml": storageclasses.storage.k8s.io "elastic-local-volume" already exists
Error from server (AlreadyExists): error when creating "pv-3c-30g.yaml": persistentvolumes "es-data-pv-0" already exists
Error from server (AlreadyExists): error when creating "pv-3c-30g.yaml": persistentvolumes "es-data-pv-1" already exists
Error from server (AlreadyExists): error when creating "pv-3c-30g.yaml": persistentvolumes "es-data-pv-2" already exists
7

why only elasticsearch will be affected by the admission webhook

IIRC, before ECK 1.2 only Elasticsearch has an admission webhook, not other resources.

kubectl create -f .

You can use kubectl apply instead if you don't want the already exists errors.

Timeout: request did not complete within requested timeout 30s

It looks like something is still wrong. Have you deleted the webhook entirely?

8

@sebgl Thank you for your reply.
Before I used kubectl apply, it kept showing "Timeout: request did not complete within requested timeout 30s", but now it works normally.

image
image976×292 9.93 KB