Kibana Connecting To Elasticsearch Behind ALB

[immolation]

Hello! I have my EFK stack running on Kubernetes, and with the ELASTICSEARCH_URL set to http://elasticsearch.logging.svc.cluster.local:9200 it works just fine. I need Kibana to connect to an Elasticsearch running in each cluster, and first wanted to try it out with just the local cluster. I set up an ingress for Elasticsearch and a route on the ALB, and when on the Kibana pod I can run a curl and get a successful response:

bash-4.2$ curl -kv https://<load balancer url>/elasticsearch-a
< HTTP/1.1 200 OK
< Date: Wed, 27 Nov 2019 20:53:50 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 489
< Connection: keep-alive
< Server: nginx/1.15.10
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< 
{
  "name" : "es-cluster-0",
  "cluster_name" : "k8s-logs",
  "cluster_uuid" : "xxx",
  "version" : {
    "number" : "6.4.3",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "fe40335",
    "build_date" : "2018-10-30T23:17:19.084789Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

I need to use -k in order to not get an error with the self-signed cert. I set the following to try to get it to work:

elasticsearch.url: https://<load balancer url>/elasticsearch-a
elasticsearch.ssl.verificationMode: none

But I still get "plugin:elasticsearch@6.4.3 Request Timeout after 3000ms" in Kibana. I feel like I'm close, as the Kibana pod can get responses back from Elasticsearch through the ALB, but I must be missing something in the settings to get it working. Let me know if you have any suggestions and thank you!

[matw (Matthias Wilhelm)]

Hi and welcome to our community!

Could you share your kibana.yml so we can have a closer look?

Thx & Best,
Matthias

[immolation]

Absolutely and thank you!

Working kibana.yml:

kibana.yml: |
---
server.name: kibana
server.host: "0"
elasticsearch.ssl.verificationMode: none
elasticsearch.url: http://elasticsearch.logging.svc.cluster.local:9200
server.basePath: /logging

When I change the elasticsearch.url to https://<ALB>.us-gov-west-1.elb.amazonaws.com/elasticsearch-a (a working endpoint that can be hit from the Kibana pod), Kibana times out when attempting to hit Elasticsearch.

[immolation]

Additionally, I added a listener on the ALB for HTTP (80) and tried http://<ALB>.us-gov-west-1.elb.amazonaws.com:80/elasticsearch-a for the elasticsearch.url and still received the timeouts (but was still able to hit it from the Kibana pod). This tells me it isn't an issue with the SSL verification.

[immolation]

Resolved. There was an issue with how I was doing the ingress.

Ingress config that ended up working for my solution:

kind: Ingress
metadata:
  name: logging-ingress
  namespace: logging
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - http:
        paths:
          - path: /logging/?(.*)
            backend:
              serviceName: kibana
              servicePort: 5601
          - path: /elasticsearch-a/?(.*)
            backend:
              serviceName: elasticsearch
              servicePort: 9200