I have deployed Elastic Search and Kibana on GKE.
However unable to get ingress working as it seems to report Kibana service as unhealthy.
I assume this is due to the root url redirecting with http 302 and GKE LB L7 expecting status of 200 always based on reading around.
Can someone confirm this and if there is a work around provide as I have spent quiet a bit of trying to get it working and seem to be stuck with the whole thing.
Hi @rubans. Kibana is definitely supported on GKE. Have you seen our Elastic Cloud on Kubernetes documentation?
Or, if you prefer a lower level implementation, we also publish Helm charts for the Elastic Stack.
These options should be much easier than trying to create and deploy your own CRD.
Appreciate the quick reply.
Unfortunately Helm charts is not an option for a corporate install I’m doing but it’s not a bad idea to understand the components anyway.
However at present I can’t get the ingress service working as mentioned and judging by some of the posts seems an issue.
Can you comment on the specific ingress issue ?
@nickpeihl Looks like GCE LB health checks in GCP always expect a 200 and it's not possible to change this, can someone respond ?
If there is a fix for this already in chart packages then can you provide on the nature of the fix as I'm not sure it has been fixed.
Hi @rubans. I don't know enough about Kubernetes to answer further. I suggest re-posting your question in the Elastic Orchestration category. Hopefully someone there can help.
Hi @nickpeihl, it's specifically a Kibana issue even if related to Orchestration so I think the forum is relevant.
I find the whole Elastic support overly complicated a bit like the platform, I have started looking at other options in the mean time due to the lack of clear support for the product.
Hi @rubans
Apologies for your struggles, read this and wanted to use this as a learning experience for myself.
I am certainly not a K8s expert etc. but I got the ngnix-ingress controller to work just fine on GKE, I am sure I am missing some nuances somewhere.
Where is How I got GKE + ECK + ngnix-ingress-controller + TLS working.
This is what I did after a created a Vanilla GKE cluster
Of course I did all the role binding stuff etc.. etc..
I am using ECK so I applied that...
kubectl apply -f https://download.elastic.co/downloads/eck/1.4.0/all-in-one.yaml
Then per here I applied
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml
I created a cert, dns etc... etc..
Then I I applied this
kubectl apply -f es-kb-ingress.yaml
This is es-kb-ingress.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
name: d3m0
spec:
version: 7.11.1
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
name: d3m0
spec:
version: 7.11.1
count: 1
elasticsearchRef:
name: d3m0
http:
tls:
selfSignedCertificate:
disabled: true
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: kibana-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- kib.mydomain.net
# Assumes Realcert with above host in it
secretName: my-kb-cert
rules:
- host: kib.mydomain.net
http:
paths:
- path: /
backend:
serviceName: d3m0-kb-http
servicePort: 5601
And this is what I got a nice secured stack from an ingress perspective (I would secure the back end as well)
Perhaps give it a try...
I suspect I am missing something, you asked for default... hmmm I will try that next
1 Like
Hi @stephenb
This is really useful. Appreciate you taking the time to create an end to end example.
I actually managed to resolve my issue yesterday with a simple fix.
My issue was down to an error in the ingress configuration.
I originally tried the default GKE ingress which I believe is not supported with Elastic stack (specifically Kibana UI)
In that I had some wildcard paths "/*" which I copied over straight to the nginx ingress, I wasn't aware that the wild card path syntax in the same manner aren't supported with nginx ingress controller. So this led to some weird issues such repeated redirects (302).
I once I swapped to path value "/", similar to what you have in your example, it all started playing nicely.
Nice!
I am halfway through with the default... but now I have a path problem LOL. I am sure I will get it sorted, perhaps I am running into the same thing... yeah just noticed the syntax slightly different between default and ngnix... hey I always like ngnix anyways!
@stephenb
Let me know if you need any additional info, not that I'm now an expert 
or anything.
One of the issues I had is that I'm behind a heavy corporate security so I couldn't just deploy the ECK or Helm charts as it is from Elastic.
I had to download and decipher the charts as I had to rebuild the various components docker images from source then apply them. So just meant, I was never sure whether it was me doing something wrong or something inherently not supported. Anyway, really appreciate you taking time to go the extra mile to assist. Always appreciated.
1 Like
@rubans
Just FYI, I got everything to work on my side for default ingress as well. I totally understand the "Corporate Sercurity " I lived with them for many years at a financial institute. I do think that occasionally there is a race condition with setting some times while ELK / Kibana is coming up and the ingress is trying to come up. I had better luck letting ECK / ELK come all the way up and then applying the ingress... if I have time I will do another one of my little How Tos.. Thanks for the co-adventure!
@stephenb do you have your deployed final conigs?
Especially interested in the ingress config to see the reason why I couldn't get it working.
Also, I raised an issue with the latest Helm Chart and ingress package which also exhibited the same issue.
@stephenb can you provide an example of gke ingress manifest? from all the elastic documentation I read, it's not possible to get it working without nginx so curious to see how you done this.
Apologies I have had some other items on my plate.
Remember I did this via ECK not Helm, I am an expert in neither, but I did this all very vanilla.
So basically following at the steps above here is what I deployed.
I did notice sometimes seems like perhaps there is a race condition or something so I deployed the elasticsearch and kibana separate and let it all come up first. Then I applied the the ingress so.
kubectl -f es-kb.yml
# elasticsearch kibana deploy with ECK
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
name: d3m0
spec:
version: 7.11.1
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
name: d3m0
spec:
version: 7.11.1
count: 1
elasticsearchRef:
name: d3m0
http:
tls:
selfSignedCertificate:
disabled: true
Then I let is all come up... then I applied the ingress and of course I had real certs.
It can take a while to come up... With no annotations it uses the default GKE ingress. I have pretty healthy privileges in this project.
kubectl apply -f ingress.yml
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: kibana-ingress
# annotations:
# kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- kbeck.mydomain.net
# Assumes Realcert with above host in it
secretName: my-kb-cert
rules:
- host: kbeck.mydomain.net
http:
# NOTE
# seems like for default ingress path: /*
# for ngnix ingress path: /
paths:
- path: /*
backend:
serviceName: d3m0-kb-http
servicePort: 5601
kubectl describe ingress
Name: kibana-ingress
Namespace: default
Address: 35.244.190.155
Default backend: default-http-backend:80 (10.20.1.5:8080)
TLS:
my-kb-cert terminates kbeck.mydomain.net
Rules:
Host Path Backends
---- ---- --------
kbeck.bvader.net
/* d3m0-kb-http:5601 (10.20.2.2:5601)
Annotations: ingress.kubernetes.io/backends: {"k8s-be-31576--fddf248ac95c1118":"HEALTHY","k8s1-fddf248a-default-d3m0-kb-http-5601-5123ae18":"HEALTHY"}
.....@stephenb thanks for sharing, Yes, looks all healthy using default ingress.
Wonder what ECK does different to when manually setting up.
The underlying issue I understood it was the issue with Kibana UI doing a 302 on root path conflicts with GCP health checks that are created which I read always require a 200.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.