Possible problem with Elasticsearch readinessProbe?

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hey all, I am trying to setup a google cloud Ingress object to route traffic to the Elasticsearch service in ECK, but it appears as though the Ingress object attempts the health check using the HTTP protocol, instead of HTTPS. I was under the impression that the readinessProbe defined in the operator overrode the default check at / looking for 200

Here is my toy ES Config:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  http:
service:
  spec:
    type: LoadBalancer
  version: 7.5.2
  nodeSets:
  - name: default
count: 1
config:
  node.master: true
  node.data: true
  node.ingest: true
  node.store.allow_mmap: false

Here is the accompanying ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: elastic-ingress
  annotations: 
    kubernetes.io/ingress.global-static-ip-name: name-of-my-address
    kubernetes.io/ingress.allow-http: "true"
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: quickstart-es-http
          servicePort: 9200

A few minutes after I start this ingress, I get the following in the cloud console:

2020-01-29_18-29
2020-01-29_18-291054×584 46.4 KB

And in my elasticsearch pod logs, I get the following warnings, which seem to indicate that the health checks are being rejected:

{"type": "server", "timestamp": "2020-01-30T02:29:14,832Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "quickstart", "node.name": "quickstart-es-default-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.0.23:9200, remoteAddress=/10.128.0.3:36878}", "cluster.uuid": "Oxno1fnRTluho2wlekONMA", "node.id": "DtOKi0UcS6K8NyAwDtsryA"  }

Is there some simple way to resolve this?

Cheers

2

The google docs are a little confusing. The health checks section makes it seem like it requires an http check (where we use an exec check). But the sections on https betwen the LB and the service and disabling HTTP seem to indicate that in this case (where you want HTTPS from the client to the LB to the Elasticsearch service), you would want to add these annotations:
cloud.google.com/app-protocols: '{"http":"HTTPS"}'
kubernetes.io/ingress.allow-http: "false"

It's not 100% clear that the latter will keep it from using HTTP health checks. I suspect it would start using https checks, but it will try and access it without authenticating which will still error out.

3

Thanks for the response!

No such luck unfortunately. After modifying the elasticsearch resource to add this annotation

metadata:
  name: quickstart
  annotations:
    cloud.google.com/app-protocols: '{"http":"HTTPS"}'

and setting the ingress to not allow HTTP, the health checks that are created still have the protocol set to http. I will wait on this: https://github.com/elastic/cloud-on-k8s/issues/2489

I have the option of also just using a LoadBalancer http service, but then I'm stuck either relying on the self-signed cert, or a self-managed cert, where I want to use the google ManagedCertificate.

4

The annotation has to be on the service template and I had to enable anonymous access for the health check to pass (I used the monitoring user role but this could be restricted further) :

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.5.2
  http:
    service:
      metadata:
        annotations:
          cloud.google.com/app-protocols: '{"https":"HTTPS"}'
      spec: 
        type: NodePort
  nodeSets:
  - name: default
    count: 3
    config:
      xpack.security.authc:
        anonymous:
          username: anonymous_user 
          roles: monitoring_user