Kibana Connecting To Elasticsearch Behind ALB

Hello! I have my EFK stack running on Kubernetes, and with the ELASTICSEARCH_URL set to http://elasticsearch.logging.svc.cluster.local:9200 it works just fine. I need Kibana to connect to an Elasticsearch running in each cluster, and first wanted to try it out with just the local cluster. I set up an ingress for Elasticsearch and a route on the ALB, and when on the Kibana pod I can run a curl and get a successful response:

bash-4.2$ curl -kv https://<load balancer url>/elasticsearch-a
< HTTP/1.1 200 OK
< Date: Wed, 27 Nov 2019 20:53:50 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 489
< Connection: keep-alive
< Server: nginx/1.15.10
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< 
{
  "name" : "es-cluster-0",
  "cluster_name" : "k8s-logs",
  "cluster_uuid" : "xxx",
  "version" : {
    "number" : "6.4.3",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "fe40335",
    "build_date" : "2018-10-30T23:17:19.084789Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

I need to use -k in order to not get an error with the self-signed cert. I set the following to try to get it to work:

elasticsearch.url: https://<load balancer url>/elasticsearch-a
elasticsearch.ssl.verificationMode: none

But I still get "plugin:elasticsearch@6.4.3 Request Timeout after 3000ms" in Kibana. I feel like I'm close, as the Kibana pod can get responses back from Elasticsearch through the ALB, but I must be missing something in the settings to get it working. Let me know if you have any suggestions and thank you!

Hi and welcome to our community! :wave:

Could you share your kibana.yml so we can have a closer look?

Thx & Best,
Matthias

Absolutely and thank you!

Working kibana.yml:

kibana.yml: |
---
server.name: kibana
server.host: "0"
elasticsearch.ssl.verificationMode: none
elasticsearch.url: http://elasticsearch.logging.svc.cluster.local:9200
server.basePath: /logging

When I change the elasticsearch.url to https://<ALB>.us-gov-west-1.elb.amazonaws.com/elasticsearch-a (a working endpoint that can be hit from the Kibana pod), Kibana times out when attempting to hit Elasticsearch.

Additionally, I added a listener on the ALB for HTTP (80) and tried http://<ALB>.us-gov-west-1.elb.amazonaws.com:80/elasticsearch-a for the elasticsearch.url and still received the timeouts (but was still able to hit it from the Kibana pod). This tells me it isn't an issue with the SSL verification.

Resolved. There was an issue with how I was doing the ingress.

Ingress config that ended up working for my solution:

kind: Ingress
metadata:
  name: logging-ingress
  namespace: logging
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - http:
        paths:
          - path: /logging/?(.*)
            backend:
              serviceName: kibana
              servicePort: 5601
          - path: /elasticsearch-a/?(.*)
            backend:
              serviceName: elasticsearch
              servicePort: 9200
1 Like