502 Bad Gateway when callback from proxy

[crimson_riot]

I am using oauth2 proxy to authenticate the user through google and then the authenticated user should connect to kibana which can be accessed via http://localhost:5601. However after authentication success (as mentioned in the 1st log), it gives 502 Bad gateway: There was a problem connecting to the upstream server.. Any ideas whats the problem here ?

The oauth2-proxy logs are looking like this:

10.20.51.169:5475- user@example.com[2022/05/10 11:12:40] [AuthSuccess] Authenticated via OAuth2: Session{email:user@example.com user:656549595959595 PreferredUsername: token:true id_token:true created:2022-05-10 11:12:40.385971851 +0000 UTC m=+2147.975924036 expires:2022-05-10 12:12:39.385971851 +0000 UTC m=+5746.975924036 refresh_token:true}

10.20.51.169:5475  - - [2022/05/10 11:12:40] kibana.sandbox.k8s.example.com GET - "/oauth2/callback?state=fefef5awef5aew:/&code=4/6a5wf650aw6f56we6f56aew6f5a60fwe56af5fa2ew6f0ef=email%20profile%20https://www.googleapis.com/auth/userinfo.profile%20https://www.googleapis.com/auth/userinfo.email%20openid&authuser=0&hd=example.com&prompt=consent" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36" 302 24 0.163

[2022/05/10 11:12:40] [error_page.go:93] Error proxying to upstream server: EOF

10.20.51.169:5475 - fawef-awef-awef-awef-FE - user@example.com [2022/05/10 11:12:40] kibana.sandbox.k8s.example.com GET / "/" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36" 502 2163 0.001

I am using ECK operator and the kibana.yml file is looking like this:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.2.0
  http:
    service:
      spec:
        type: LoadBalancer
        ports:
        - name: https
          port: 443
          targetPort: 3000
      metadata:
        annotations:
          # Note that the backend talks over HTTP.
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
          # TODO: Fill in with the ARN of your certificate.
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:***
          # Only run SSL on the port named "https" below.
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    tls:
      selfSignedCertificate:
        subjectAltNames:
        - dns: kibana.sandbox.k8s.example.com      
  count: 1
  elasticsearchRef:
    name: kube-es
  podTemplate:
    spec:
      containers:
      - name: kibana
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2.5Gi
            cpu: 2
        ports:
          - containerPort: 5601
            name: http
            protocol: TCP
      - name: kibana-proxy
        image: 'quay.io/oauth2-proxy/oauth2-proxy:latest'
        imagePullPolicy: IfNotPresent
        args:
          - --cookie-secret=sergawergawgr4agrgargrgarg=
          - --client-id=872911544486-otlttds9nh9t6h2ifovba0kcd6sa3seb.apps.googleusercontent.com
          - --client-secret=iijIIIIJIIE_EDEWQID_DQWDWQD
          - --upstream=http://localhost:5601
          - --email-domain=example.com
          - --footer=-
          - --http-address=http://:3000
          - --redirect-url=https://kibana.sandbox.k8s.example.com/oauth2/callback
        ports:
          - containerPort: 3000
            name: http
            protocol: TCP
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 0.5
            memory: 256Mi

Let me know if anything is needed. Thanks

[michael.morello]

    tls:
      selfSignedCertificate:
        subjectAltNames:
        - dns: kibana.sandbox.k8s.example.com 

This adds kibana.sandbox.k8s.example.com to the Kibana certificate. However I'm not sure it makes sense given that Kibana should be accessed on its localhost interface by the proxy, not through this fqdn? (I'm not familiar with the oauth2 proxy, maybe I'm missing something)

        ports:
          - containerPort: 5601
            name: http
            protocol: TCP

If Kibana is supposed to listen on a secured endpoint I would expected https:

    ports:
    - containerPort: 5601
      name: https
      protocol: TCP

Note that this is automatically managed by the operator, any reason to add it manually in the Kibana spec?

Also I would expect https in the upstream URL:

         - --upstream=http://localhost:5601

         - --upstream=https://localhost:5601

Is there any relevant logs from the Kibana container?

[crimson_riot]

Hi Michael,

Thank you for your response and suggestions. It works now after I disabled the TLS . Well, I just wanted to have user authentication on secure channel (user to kibana-proxy) and then I don't care if the internal communication between kibana proxy and kibana is secure or not. I disabled TLS and it worked. Thanks

    tls:
      selfSignedCertificate:
        disabled: true