Kibana can't connect to ES having (openstack) LoadBalancer setting

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I want to use ES and Kibana on LoadBalancer Env. So, i added load balancer options to ES yaml as below.

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: lb
spec:
  http:
    service:
       metadata:
         annotations:
           service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
       spec:
          externalTrafficPolicy: Local
          type: LoadBalancer
...

And then, i executed Kibana yaml(default) as below.

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: test
spec:
  version: 6.8.0
  count: 1
  elasticsearchRef:
    name: "lb"
  podTemplate:
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 1Gi
            cpu: 1

On this settings, kibana coudln't be executed normally having error as below.

kubernetes error
Warning Unhealthy 2m43s (x179 over 32m) kubelet, dkosv3-ingress-test-worker-4 Readiness probe failed: HTTP probe failed with statuscode: 503

docker error
{"type":"log","@timestamp":"2019-11-05T22:57:32Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://lb-es-http.default.svc:9200/"} {"type":"log","@timestamp":"2019-11-05T22:57:32Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-11-05T22:57:32Z","tags":["warning","task_manager"],"pid":1,"message":"PollError No Living connections"}

So, i want to know that what options is needed to use kibana for es having load balancer settings.

2

Hey @JuYeong_Park,

A few questions to better understand the problem:

  • Are you able to access the Elasticsearch API (without Kibana)?
  • Are your Elasticsearch Pods up and running?
  • Do things work without the LoadBalancer service?
1 Like
3

@sebgl

Thank you for your response.

  1. Are you able to access the Elasticsearch API (without Kibana)?
    Yes. I can access Elasticsearch on EXTERNAL-IP.

curl -u "elastic:$PASSWORD" -k "http://EXTERNAL_IP:9200"

responds right result.

  1. Are your Elasticsearch Pods up and running?
    Yes. All pods and service are running status.

  2. Do things work without the LoadBalancer service?
    Yes. When i used Elasticsearch without LB service, kibana could access Elasticsearch normally. But, when i use Elasticsearch with LB, then kibana pod has a status like below image.

image

Below error saying, i think that Kibana can't find accurate host of "lb-es-http.default.svc". Maybe mapping error?

{"type":"log","@timestamp":"2019-11-05T22:57:32Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://lb-es-http.default.svc:9200/"}

4

Interesting. According to your experience it looks like Openstack internal LoadBalancer prevents the internal service DNS from resolving. Which would make sense, I guess, if you want another "internal" endpoint.

If that helps, you should be able to keep the existing Elasticsearch service with default options, so Kibana can reach it normally. Then create your own additional Service to target Elasticsearch Pods the way you want:

apiVersion: v1
kind: Service
metadata:
  name: your-service-name-here
  annotations:
    service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  ports:
  - name: https
    port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: elasticsearch-sample
1 Like
5

@sebgl

Thank you for your advice.
As your advice, i seperated my yaml file into 1) Elasticsearch search with default options and 2) Service for the LB settings like you wrote above.

And, i finally can use Elasticsearch on the LB and kibana.

Thank you again for your kind advice.

1 Like