SSL error in Kibana : error:1408F09C:SSL routines:ssl3_get_record:http request

Hi,
I have ELK stack running in Openshift 3.11, now need to integrate Kibana with AD using openshift OAuth proxy. After adding the Oauth container with Kibana deployment was able to get the openshift login page, but once I login getting below errors which I presume is occuring when kibana trying to connect to ES. Appreciate any help regards with certificates needed to be used and how, custom/openshift/ES.

"tags":["connection","client","error"],"pid":8,"level":"error","error":{"message":"139735490852736:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 139735490852736:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"139735490852736:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}

ES yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: bcnc-logging
spec:
  version: 7.6.1
  nodeSets:
  - name: default
    resources:
      limits:
        memory: 8Gi
#        memory: 4Gi
        cpu: 1
    env:
    - name: ES_JAVA_OPTS
      value: "-Xms2g -Xmx2g"
    count: 3
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app: elasticsearch
      spec:
        accessModes: [ "ReadWriteOnce" ]
#        storageClassName: nas
        storageClassName: aws-storage
        resources:
          requests:
            storage: 10Gi
    
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: elasticsearch
  namespace: bcnc-logging
spec:
#  host: elasticsearch-bcnc-logging.ip-10-149-8-11.dev.aws.bcbsnc.com
  host: elasticsearch-bcnc-logging.apps2.lab.tpt01.rdu.bcbsnc.com
  tls:
    termination: passthrough # Elasticsearch is the TLS endpoint
    insecureEdgeTerminationPolicy: Redirect
  to:
    kind: Service
    name: elasticsearch-es-http

Kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: bcnc-logging
spec:
  version: 7.6.1
  count: 1
  # config:
  #   xpack.security.enabled: true
#     xpack.security.audit.enabled: true
#     elasticsearch.username: elastic
# #    elasticsearch.ssl.certificateAuthorities: /etc/certs/ca.crt
#     elasticsearch.ssl.certificate: /etc/tls/private/tls.crt
#     elasticsearch.ssl.key: /etc/tls/private/tls.key
#     elasticsearch.requestHeadersWhitelist: [ 'authorization', 'X-Proxy-Remote-User', 'x-forwarded-for', 'x-forwarded-access-token' ]
#     xpack.monitoring.kibana.collection.enabled: false
#     xpack.monitoring.ui.container.elasticsearch.enabled: false
#     xpack.reporting.kibanaServer.hostname: 0.0.0.0
  elasticsearchRef:
    name: "elasticsearch"
  podTemplate:
    spec:
      automountServiceAccountToken: true
      containers:
      - name: kibana
        resources:
          limits:
            memory: 2Gi
            cpu: 1
      - name: kibana-proxy
        image: 'registry.redhat.io/openshift3/oauth-proxy:latest'
        imagePullPolicy: IfNotPresent
        args:
          - -provider=openshift
          - -https-address=:3000
          - -http-address=
          - -email-domain=*
          - -upstream=http://localhost:5601
          - -openshift-service-account=bcnc-logging-sa
          - -cookie-secret-file=/etc/proxy/secret/session_secret
          - -tls-cert=/mnt/elastic-internal/http-certs/tls.crt
          - -tls-key=/mnt/elastic-internal/http-certs/tls.key
          - -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
          - -openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          - -openshift-ca=/usr/share/kibana/config/elasticsearch-certs/ca.crt
          - -skip-provider-button=true
        env:
          - name: OAP_DEBUG
            value: 'False'
          - name: OCP_AUTH_PROXY_MEMORY_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: kibana-proxy
                divisor: '0'
                resource: limits.memory
        ports:
          - name: oaproxy
            containerPort: 3000
            protocol: TCP
        resources:
          limits:
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 256Mi
        volumeMounts:
          - mountPath: /usr/share/kibana/config/elasticsearch-certs
            name: elasticsearch-certs
          - mountPath: /mnt/elastic-internal/http-certs
            name: elastic-internal-http-certificates
          - mountPath: /etc/proxy/secret
            name: kibana-proxy
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccountName: bcnc-logging-sa
      terminationGracePeriodSeconds: 30
      volumes:
        - name: kibana-proxy
          secret:
            defaultMode: 420
            secretName: bcnc-logging-proxy

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    serviceaccounts.openshift.io/oauth-redirectreference.kib: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"kibana-kb-http"}}'
  name: bcnc-logging-sa
  namespace: bcnc-logging

---
apiVersion: v1
data:
  session_secret: XXXXXXXXXXXXXX
kind: Secret
metadata:
  name: bcnc-logging-proxy
  namespace: bcnc-logging
  type: Opaque

---
apiVersion: v1
kind: Service
metadata:
  annotations:
     service.alpha.openshift.io/serving-cert-secret-name: kibana-tls
  labels:
    common.k8s.elastic.co/type: kibana
    kibana.k8s.elastic.co/name: kibana
  name: kibana-kb-http
  namespace: bcnc-logging
spec:
  ports:
  # - name: https
  #   port: 5601
  #   protocol: TCP
  #   targetPort: 5601
  - name: oaproxy
    port: 443
    protocol: TCP
    targetPort: oaproxy
  selector:
    common.k8s.elastic.co/type: kibana
    kibana.k8s.elastic.co/name: kibana
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: v1
kind: Route
metadata:
  name: kibana
  namespace: bcnc-logging
spec:
  host: kibana-logging.ip-10-149-8-11.dev.com
  port:
    targetPort: oaproxy
  tls:
    termination: passthrough # Kibana is the TLS endpoint
    insecureEdgeTerminationPolicy: Redirect
  to:
    kind: Service
    name: kibana-kb-http

Hi @Mastana_Guru,

Please see for setting up certificates: https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-tls-certificates.html. Let us know if this helps.

Thanks,
Liza

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.