HI @azasypkin
- I have reverted back the server basepath changes and attached the logs.
Retried on private tab but makes no difference and still an issue.
- My ingress now looks like this
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-efk
namespace: kube-logging-dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/use-regex: "true"
# nginx.ingress.kubernetes.io/rewrite-target: /
ingress.gcp.kubernetes.io/pre-shared-cert: "foo"
kubernetes.io/ingress.class: "nginx"
# kubernetes.io/ingress.allow-http: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: monitor.foo.com
http:
paths:
- backend:
serviceName: kibana
servicePort: 5601
path: /*
- host: monitor.foo.com
http:
paths:
- backend:
serviceName: elasticsearch
servicePort: 9200
path: /elasticsearch(/|$)(.*)
-
Deployment looks like this
apiVersion: v1
kind: ConfigMap
metadata:
name: kibana-configmap
data:
kibana.yml: |
server.name: kibana
server.host: 0.0.0.0
#server.rewriteBasePath: true
#server.basePath: /kibana
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
X-Pack security credentials
elasticsearch.username: elastic
elasticsearch.password: changeme
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kibana
namespace: kube-logging-dev
labels:
app: kibana
spec:
serviceName: kibana
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
nodeSelector:
workloadType: "system"
containers:
- name: kibana
image: gcr.io/foo/kibana/kibana:7.8.1-v2
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
readinessProbe:
httpGet:
scheme: HTTP
path: /
port: 5601
env:
- name: ELASTICSEARCH_LOGGING_SERVICE_HOST
value: "elasticsearch"
- name: ELASTICSEARCH_LOGGING_SERVICE_PORT
value: "9200"
ports:
- containerPort: 5601
name: ui
protocol: TCP
securityContext:
runAsUser: 999
runAsGroup: 999
volumeMounts:
- name: config-volume
mountPath: /usr/share/kibana/config
volumes:
- name: config-volume
configMap:
name: kibana-configmap
items:
- key: kibana.yml
path: kibana.yml
4) The issue with the relative path /elasticsearch is not an issue for now as I'm using local cluster connection and works as verified by Kibana logs.
5) I can't attach any files to the post but let me know how to post the HAR and Kibana logs.
Here's a sample of difference between internal kube proxy health check one and my external request:
{"type":"response","@timestamp":"2021-02-18T09:36:46Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/spaces/enter","method":"get","headers":{"host":"192.168.160.33:5601","user-agent":"kube-probe/1.18+","referer":"http://192.168.160.33:5601/","accept-encoding":"gzip","connection":"close"},"remoteAddress":"192.168.194.31","userAgent":"192.168.194.31","referer":"http://192.168.160.33:5601/"},"res":{"statusCode":302,"responseTime":39,"contentLength":9},"message":"GET /spaces/enter 302 39ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:46Z","tags":,"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"192.168.160.33:5601","user-agent":"kube-probe/1.18+","referer":"http://192.168.160.33:5601/spaces/enter","accept-encoding":"gzip","connection":"close"},"remoteAddress":"192.168.194.31","userAgent":"192.168.194.31","referer":"http://192.168.160.33:5601/spaces/enter"},"res":{"statusCode":200,"responseTime":80,"contentLength":9},"message":"GET /app/kibana 200 80ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:47Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"monitor.foo.dev.gcp.cloud.uk","x-request-id":"bd47e538951bfb95f77c38613d2a6224","x-real-ip":"192.168.194.28","x-forwarded-for":"192.168.194.28","x-forwarded-host":"monitor.foo.dev.gcp.cloud.uk","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","pragma":"no-cache","cache-control":"no-cache","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"192.168.161.14","userAgent":"192.168.161.14"},"res":{"statusCode":302,"responseTime":22,"contentLength":9},"message":"GET / 302 22ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:47Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"monitor.foo.dev.gcp.cloud.uk","x-request-id":"e3ddaeadf1672adac660911103f958ce","x-real-ip":"192.168.194.28","x-forwarded-for":"192.168.194.28","x-forwarded-host":"monitor.foo.dev.gcp.cloud.uk","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","pragma":"no-cache","cache-control":"no-cache","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"192.168.161.14","userAgent":"192.168.161.14"},"res":{"statusCode":302,"responseTime":22,"contentLength":9},"message":"GET / 302 22ms - 9.0B"}
NB : I have tried to obfuscate any senstive info from the logs
Pls let me know next steps as it makes no sense why this wouldn't work.