Failed to ensure load balancer for service default/quickstart-es-http

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello,

I'm trying to deploy quickstart to our Kubernetes Cluster and get stuck with LoadBalancer.

Name:                     quickstart-es-http
Namespace:                default
Labels:                   common.k8s.elastic.co/type=elasticsearch
                          elasticsearch.k8s.elastic.co/cluster-name=quickstart
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"creationTimestamp":"2020-03-24T10:36:31Z","labels":{"common.k8s.elastic....
Selector:                 common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=quickstart
Type:                     LoadBalancer
IP:                       10.247.213.185
Port:                     https  9200/TCP
TargetPort:               9200/TCP
NodePort:                 https  30448/TCP
Endpoints:                172.16.0.51:9200
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                      Age                   From                Message
  ----     ------                      ----                  ----                -------
  Normal   EnsuringLoadBalancer        7d17h (x139 over 8d)  service-controller  Ensuring load balancer
  Normal   UpdatedLoadBalancer         22m                   service-controller  Updated load balancer with new pods
  Normal   EnsuringLoadBalancer        2m49s (x10 over 23m)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  2m49s (x10 over 23m)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service default/quickstart-es-http: service loadbalancer IP is invalid

Can you please advise where to start digging in? Thanks a lot in advance

2

Hey @tillias, can you provide the Elasticsearch yaml manifest you are using?
What's your Kubernetes environment? Are you running on a cloud provider?
It looks like it complains about the LoadBalancer IP, which is not something ECK manages.

3

Yes, here is es-sample.yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.6.1
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: nfs-rw

kubectl get services
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes              ClusterIP   10.247.0.1       <none>        443/TCP    12m
quickstart-es-default   ClusterIP   None             <none>        <none>     2m56s
quickstart-es-http      ClusterIP   10.247.224.180   <none>        9200/TCP   2m57s

kubectl get -o yaml --export svc/quickstart-es-http

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: quickstart
  name: quickstart-es-http
  ownerReferences:
  - apiVersion: elasticsearch.k8s.elastic.co/v1
    blockOwnerDeletion: true
    controller: true
    kind: Elasticsearch
    name: quickstart
    uid: e6df1a29-7430-11ea-9768-fa163efcd30d
  selfLink: /api/v1/namespaces/default/services/quickstart-es-http
spec:
  ports:
  - name: https
    port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: quickstart
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

The last one is created automatically by Kubernetes Operator as far as I understand. In order to make it using load balancer I simply modify type: ClusterIP for quickstart-es-http to type:LoadBalancer

4

As Seb requested, can you provide more detail about your k8s environment please? Following the instructions here I was unable to reproduce your issue on GKE. It appears that it may be something environment or provider specific.

  Type    Reason                Age    From                Message
  ----    ------                ----   ----                -------
  Normal  Type                  3m51s  service-controller  ClusterIP -> LoadBalancer
  Normal  EnsuringLoadBalancer  3m51s  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   3m1s   service-controller  Ensured load balancer
5

Hi Anya,

Sorry that I didn't mentioned it before, we're using Open Telekom Cloud as a cloud provider and managed Kubernetes Cluster there. I agree this is provider specific issue, because in GKE it works fine. Is there any possibility to configure / tune Load Balancer settings like we did with storage class in this topic Pod has unbound immediate PersistentVolumeClaims ?

6

Based on the documentation at https://docs.otc.t-systems.com/en-us/usermanual2/cce/cce_01_0014.html#cce_01_0014__section07201626201720, something like the following might work for you. Please note that I don't have a way of testing it myself so this is mostly guesswork.

You'll probably need to change some of the annotation values (subnet-id, bandwidth_name) to match your environment as well. Refer to the documentation or your system administrator.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.6.1
  http:
    service:
      metadata:
        annotations:
          service.protal.kubernetes.io/type: LoadBalancer
          kubernetes.io/elb.class: union
          kubernetes.io/elb.subnet-id: eff413e6-e6df-43e6-b586-ee36351d676d
          kubernetes.io/elb.autocreate: '{"type":"public","bandwidth_name":"cce-bandwidth-1548668872423","bandwidth_chargemode":"traffic","bandwidth_size":10,"bandwidth_sharetype":"PER","eip_type":"5_bgp"}'
      spec:
        type: LoadBalancer
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
7

Thaks for the reply.

If use this yaml for cluster creation then quickstart-es-http is still created with ClusterIP

kubectl get services
NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes              ClusterIP   10.247.0.1     <none>        443/TCP    10m
quickstart-es-default   ClusterIP   None           <none>        <none>     4s
quickstart-es-http      ClusterIP   10.247.76.42   <none>        9200/TCP   5s
8

That's odd. Could be an issue with the indentation of the YAML. Assuming you have it saved as otc.yaml, make sure the following prints "LoadBalancer":

kubectl apply -f otc.yaml --dry-run -o=jsonpath='{.spec.http.service.spec.type}{"\n"}'

The other possibility is that some configuration in OTC is preventing you from creating load balancers. Probably worth checking with OTC support to see if that's the case.