Public SSL'ed access with Ingress not working

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
4

Hello again!

I've figured out what was broken. It turns out, that the default readinessProbe for the kibana pod looks like this:

 kubectl get deployment quickstart-kibana -o jsonpath='{.spec.template.spec.containers[*].readinessProbe}'
Readiness:  http-get http://:5601/ delay=10s timeout=5s period=10s #success=1 #failure=3

Unfortunately, this returns a 302 to /login?next=%2f as evidenced by the pod logs:
podlogs

Under kubernetes, readinessProbes are allowed to return anything from 200 >= response_code < 400. Unfortunately, when creating an Ingress on GCP (the default Http Cloud Load Balancer), it automatically creates a GCP http health check against the readinessProbe (in this case /), which must return 200. See here

The solution to this just ended up being a fix to the health check itself in this case, since I'm unable to modify the readinessProbe of a running pod. In order to fix the associated health-check, I had to update the request path to point to the easiest thing that returns a 200, the login page:

First, find the health check in question.

 gcloud compute health-checks list
NAME                            PROTOCOL
k8s-be-32228--6ac986474e4810bf  HTTP

then update the http health check to point to the endpoint which actually returns 200:

  gcloud compute health-checks update http k8s-be-32228--6ac986474e4810bf --request-path=/login\?next=%2F

Then verify the health-check has been updated by checking the request path

 gcloud compute health-checks describe k8s-be-32228--6ac986474e4810bf
checkIntervalSec: 70
creationTimestamp: '2019-07-11T10:43:46.547-07:00'
description: Kubernetes L7 health check generated with readiness probe settings.
healthyThreshold: 1
httpHealthCheck:
  port: 32228
  proxyHeader: NONE
  requestPath: /login?next=%2F
id: '8253387266199309245'
kind: compute#healthCheck
name: k8s-be-32228--6ac986474e4810bf
selfLink: [redacted]
timeoutSec: 5
type: HTTP
unhealthyThreshold: 10

5 minutes after this, the health checks came alive and my site is now routing correctly using default GCP ingress.

It may be worthwhile to change the default readinessProbe requestPath for the kibana pods. I'm not sure how Azure/AWS handles health checks, but for GCP at least this operator doesn't work out of the box with ingress.