Getting error while enabled tls

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Enabled tls certs in elasticsearch and kibana.Using custom certificates and passed the secrets in elasticsearch.yaml. Elasticsearch pods are running. But kibana pod is not in running state. Below are the kibana logs:
{"type":"log","@timestamp":"2020-12-02T15:00:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-12-02T15:00:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http..svc:9200/"}
{"type":"log","@timestamp":"2020-12-02T15:00:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-12-02T15:00:18Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http.-.svc:9200/"}

Is there anything else to pass the secrets in kibana.yaml as well.

2

Hello,

I am not expert in Elastic but I think it's better if you share your elasticsearch and kibana configuration.
I share with you my TLS configuration in my cluster, it may maybe help you

elasticsearch.yml

#=============== Encryption between Elasticsearch nodes ===============
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: node.key
xpack.security.transport.ssl.certificate: node.crt
xpack.security.transport.ssl.certificate_authorities: [ "ca.crt" ]

#=============== Encryption between Elastic and kibana ===============
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12
xpack.security.http.ssl.truststore.path: http.p12

Kibana.yml

 #================== Encryption between kibana and Browser===================
server.ssl.enabled: true
server.ssl.certificate: "KIBANA.crt"
server.ssl.key: "KIBANA.key"
elasticsearch.ssl.certificateAuthorities: [ "elasticsearch-ca.pem"]
elasticsearch.ssl.verificationMode: full
3

Thank you @TheHunter1
storing certs in a kubernetes secret and passing the secret into elasticsearch and kibana yaml files as below

spec:
    http:
      tls:
        certificate:
          secretName: <secret-name>

kibana pod is running. but connection to elasticsearch is lost

4

I am sorry I don't have experience with kubernetes, I let experienced members answer your question

Best regards

5

Can anyone answer to this?

6

Hi,

Could you describe your Secret and the certificates: are they self signed, from a private CA or issued by a well know authority ?
Could you also provide us the Kibana full logs and the manifests (es/kb)?

Thanks

7

I am using private CA. Below is the output for secret:

[root@lvndev012336 ~]# kubectl -n es-crt describe secret <secret-name>
Name:         ####
Namespace:    es-crt
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
tls.crt:  1964 bytes
tls.key:  1679 bytes

Below are the logs for kibana:

{"type":"log","@timestamp":"2020-12-10T04:49:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http.es-crt.svc:9200/"}
{"type":"log","@timestamp":"2020-12-10T04:49:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-12-10T04:49:29Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http.es-crt.svc:9200/"}
{"type":"log","@timestamp":"2020-12-10T04:49:29Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-12-10T04:49:29Z","tags":["license","warning","xpack"],"pid":6,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2020-12-10T04:49:30Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http.es-crt.svc:9200/"}
{"type":"log","@timestamp":"2020-12-10T04:49:30Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}

Below are logs of operator:

{"log.level":"info","@timestamp":"2020-12-10T04:32:33.570Z","log.logger":"elasticsearch-controller","message":"Elasticsearch manifest has warnings. Proceed at your own risk. [spec.nodeSets[0].config.xpack.security.http.ssl.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[1].config.xpack.security.http.ssl.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported]","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","namespace":"es-crt","es_name":"elasticsearch-config"}
{"log.level":"info","@timestamp":"2020-12-10T04:32:34.146Z","log.logger":"keystore","message":"Secure settings secret not found","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","namespace":"es-crt","secret_name":"gcs-credentials"}
{"log.level":"info","@timestamp":"2020-12-10T04:32:34.155Z","log.logger":"zen2","message":"Ensuring no voting exclusions are set","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","namespace":"es-crt","es_name":"elasticsearch-config"}
{"log.level":"info","@timestamp":"2020-12-10T04:32:34.292Z","log.logger":"migrate-data","message":"Setting routing allocation excludes","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","namespace":"es-crt","es_name":"elasticsearch-config","value":"none_excluded"}
{"log.level":"info","@timestamp":"2020-12-10T04:32:35.384Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","iteration":52,"namespace":"es-crt","es_name":"elasticsearch-config","took":1.815017325}
{"log.level":"info","@timestamp":"2020-12-10T04:42:33.979Z","log.logger":"kibana-controller","message":"Starting reconciliation run","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","iteration":11,"namespace":"es-crt","kibana_name":"kibana-config"}
{"log.level":"info","@timestamp":"2020-12-10T04:42:33.980Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"es-crt","name":"kibana-config-kb-http"}
{"log.level":"info","@timestamp":"2020-12-10T04:42:34.001Z","log.logger":"kibana-controller","message":"Ending reconciliation run","service.version":"1.1.0-29e7447f","service.type":"eck","ecs.version":"1.4.0","iteration":11,"namespace":"es-crt","kibana_name":"kibana-config","took":0.02179661}
8

Can anyone answer to the above error?

9

Can you provide your full Elasticsearch & Kibana yaml manifests?
Are you able to connect to Elasticsearch? Are your Elasticsearch Pods running?
Is there anything in Elasticsearch Pod logs worth mentioning about certificates and/or failed connections?

10

elasticsearch.yaml:

  apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    name: elasticsearch-config
  spec:
    http:
      tls:
        certificate:
          secretName: <custom-cert>
    version: 7.4.0
    # Following secure settings is to add gcs credentials obtained from service account for  backup purpose
    secureSettings:
     - secretName: gcs-credentials
    # Following auth filed is for configuring filerealm user and roles using secrets
    auth:
      fileRealm:
      - secretName: user-management-filerealm-secret
      roles:
      - secretName: custom-role-secret
    nodeSets:
    - name: master
      count: 1
      config:
        node.master: true
        node.data: false
        node.ingest: false
        node.ml: true
        # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
        node.store.allow_mmap: false

        # for release 7.4.0 and 7.6.0 uncomment following line for filerealm config
        xpack.security.authc.realms.file.file1.order: 0
        http.compression: true
        http.compression_level: 9
      podTemplate:
        metadata:
          labels:
            # additional labels for pods
            master: node
        spec:
          initContainers:
          - name: install-plugins
            command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install --batch repository-gcs
          containers:
          - name: elasticsearch
            # specify resource limits and requests
            resources:
              limits:
                memory: 3Gi
                cpu: 1
            env:
            - name: ES_JAVA_OPTS
              value: "-Xms2g -Xmx2g"   
      volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 200Gi
          storageClassName: standard    
    - name: data-ingest
      count: 1
      config:
        node.master: false
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
        # for release 7.4.0 and 7.6.0
        xpack.security.authc.realms.file.file1.order: 0
      podTemplate:
        metadata:
          labels:
            # additional labels for pods
            data: node
        spec:
          initContainers:
          - name: install-plugins
            command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install --batch repository-gcs

kibana.yaml:

  apiVersion: kibana.k8s.elastic.co/v1
  kind: Kibana
  metadata:
    name: kibana-config
  spec:
    version: 7.4.0
    count: 1
    elasticsearchRef:
      name: "elasticsearch-config"
    http:
     tls:
      certificate:
        secretName: <custom-cert>
    podTemplate:
      metadata:
        labels:
          kibana: node
      spec:
        containers:
        - name: kibana
          resources:
            limits:
              memory: 1Gi
              cpu: 1
          readinessProbe:
            httpGet:
              scheme: HTTP
              path: "/login"
              port: 5601
11

my elasticsearch pods are running and healthy. I am able to connect with elasticsearch as well. Below are elasticsearch logs

{"type": "server", "timestamp": "2020-12-17T10:59:24,675Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch-config", "node.name": "elasticsearch-config-es-data-ingest-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/240.0.107.92:9200, remoteAddress=/240.0.46.80:43804}", "cluster.uuid": "CprNdQehSEGSI5l9Etudag", "node.id": "KmvwN0Y5SWiWph3SV6vXsw"  }
{"type": "server", "timestamp": "2020-12-17T10:59:25,119Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch-config", "node.name": "elasticsearch-config-es-data-ingest-0", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/240.0.107.92:9200, remoteAddress=/240.0.46.80:43820}", "cluster.uuid": "CprNdQehSEGSI5l9Etudag", "node.id": "KmvwN0Y5SWiWph3SV6vXsw"  }

Below are kibana pod logs

{"type":"log","@timestamp":"2020-12-17T11:00:24Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http.es-crt.svc:9200/"}
{"type":"log","@timestamp":"2020-12-17T11:00:24Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-12-17T11:00:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-config-es-http.es-crt.svc:9200/"}
{"type":"log","@timestamp":"2020-12-17T11:00:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
12

Can anyone answer to this?

13

I'm wondering if received plaintext http traffic on an https channel in Elasticsearch logs could indicate Kibana is using http instead of https to connect to Elasticsearch.

Could you paste your full Kibana resource as existing on the apiserver? It should contain some annotations that will help us understand the problem: kubectl get kibana kibana-config -o json.

14

Below is my kibana resource existing on api server:

 kubectl -n es-tls-test get kibana kibana-config -o yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  annotations:
    association.k8s.elastic.co/es-conf: '{"authSecretName":"kibana-config-kibana-user","authSecretKey":"es-tls-test-kibana-config-kibana-user","caCertProvided":true,"caSecretName":"kibana-config-kb-es-ca","url":"https://elasticsearch-config-es-http.es-tls-test.svc:9200"}'
    common.k8s.elastic.co/controller-version: 1.1.0
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kibana.k8s.elastic.co/v1","kind":"Kibana","metadata":{"annotations":{},"name":"kibana-config","namespace":"es-tls-test"},"spec":{"count":1,"elasticsearchRef":{"hosts":["https://dns:9200"],"name":"elasticsearch-config"},"image":"gcr.io/elastic-operator/kibana:7.4.0","podTemplate":{"metadata":{"labels":{"kibana":"node"}},"spec":{"containers":[{"name":"kibana","resources":{"limits":{"cpu":1,"memory":"1Gi"}}}]}},"version":"7.4.0"}}
  creationTimestamp: "2020-12-21T06:31:02Z"
  generation: 9
  name: kibana-config
  namespace: es-tls-test
  resourceVersion: "1074338814"
  selfLink: /apis/kibana.k8s.elastic.co/v1/namespaces/es-tls-test/kibanas/kibana-config
  uid: 9e4572d3-d1b8-4604-bad2-49e21d71a783
spec:
  count: 1
  elasticsearchRef:
    hosts:
    - https://dns:9200
    name: elasticsearch-config
  image: gcr.io/elastic-operator/kibana:7.4.0
  podTemplate:
    metadata:
      creationTimestamp: null
      labels:
        kibana: node
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            cpu: 1
            memory: 1Gi
  version: 7.4.0
status:
  associationStatus: Established
  availableNodes: 1
  health: green
15

Can anyone answer to this?