SSL handshake fails between Kibana/APMServer and Elasticsearch after custom certificate setting on Elasticsearch

I am also in need of an answer on this scenario @pebrc. My setup is exactly as described by @Bingu_Shim, although the issue came about for me when upgrading from 1.0.1 => 1.1.0. With 1.0.1 I am able to supply my own cert for both elastic and kibana and have them both configured with DNS. When trying with 1.1.0, I get the same errors above.

Did you ever find a solution to this?

1 Like

Can you share your config @lsnyder? The one in the OP should still work, though note that the http.tls.selfSignedCertificate and .certificate settings should not be used at the same time. The provided certificate should still take priority, but we will likely add validation to prevent this configuration in the future (as setting a custom certificate disables generation of self signed certs).

Secondly, Kibana should work with custom Elasticsearch certificates, but APM will either need the host names to match or for TLS to be disabled completely. Kibana supports validating everything in the certificate but the host name, while APM does not yet. If you can share the specifics of your issue we can try and troubleshoot it anew.

No I was not able to find the solution.

My setup does not include APM, it is just Elastic + Kibana
Here is my config for elastic and kibana:

# elastic.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-prd-uks
spec:
  auth: {}
  http:
    service:
      metadata:
        creationTimestamp: null
      spec:
        type: LoadBalancer
    tls:
      certificate:
        secretName: <my-secret-name>
  nodeSets:
  - config:
      node.attr.attr_name: attr_value
      node.data: true
      node.ingest: true
      node.master: true
      node.ml: true
      xpack.monitoring.collection.enabled: true
      xpack.security.authc.realms:
        native:
          native1:
            order: 1
    count: 3
    name: default
    podTemplate:
      metadata:
        creationTimestamp: null
        labels:
          foo: bar
      spec:
        containers:
        - env:
          - name: ES_JAVA_OPTS
            value: -Xms6g -Xmx6g
          name: elasticsearch
          resources:
            limits:
              cpu: 7350m
              memory: 11Gi
            requests:
              cpu: 7350m
              memory: 11Gi
        initContainers:
        - command:
          - sh
          - -c
          - sysctl -w vm.max_map_count=262144
          name: sysctl
          resources: {}
          securityContext:
            privileged: true
        - command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-azure
          name: install-plugins
          resources: {}
    volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
      status: {}
  secureSettings:
  - secretName: <redacted>
  - secretName: <redacted>
  transport:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
  updateStrategy:
    changeBudget: {}
  version: 7.6.2


# kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-prd-uks
  namespace: default
spec:
  count: 1
  elasticsearchRef:
    name: elasticsearch-prd-uks
  http:
    service:
      metadata: {}
      spec:
        type: LoadBalancer
    tls:
      certificate:
        secretName: <my-secret-name>
  podTemplate:
    metadata: {}
    spec: {}
  version: 7.6.2

Kibana Logs:

{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nHEAD https://elasticsearch-prd-uks-es-http.default.svc:9200/.apm-agent-configuration => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-prd-uks-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-prd-uks-es-http.default.svc:9200/_xpack => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-05-07T17:16:54Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:16:56Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:16:56Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:16:59Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:16:59Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:17:01Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}

I just tried this configuration with new released version

  • elastic operator : 1.1.0
  • Elasticsearch, Kibana : 7.6.2

And got same error as you got

Here is kibana error

2020-05-09T08:24:14.077382373Z {"type":"log","@timestamp":"2020-05-09T08:24:14Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"case\" is disabled."}
2020-05-09T08:25:29.443788733Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [37] plugins: [infra,taskManager,siem,licensing,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
2020-05-09T08:25:29.444798641Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","infra"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.447690701Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","taskManager"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.7385755Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","siem"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.739647503Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","licensing"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.744117332Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","encryptedSavedObjects"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.745404276Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":7,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
2020-05-09T08:25:29.84346468Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","code"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.844729461Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","usageCollection"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.847087285Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","metrics"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.847812528Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","canvas"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.853667359Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","timelion"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.854613642Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","features"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.855590132Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","security"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.874897033Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","apm_oss"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.875407265Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","translations"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.876092198Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","data"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.938860566Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","share"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.940414124Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","home"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.945554172Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","spaces"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.950075428Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","cloud"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.95119985Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","apm"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.955940613Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","graph"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.958959265Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","bfetch"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.965921138Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
2020-05-09T08:25:30.061930512Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","data"],"pid":7,"message":"Request error, retrying\nHEAD https://quickstart-es-http.default.svc:9200/.apm-agent-configuration => unable to verify the first certificate"}
2020-05-09T08:25:30.135797692Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","admin"],"pid":7,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
2020-05-09T08:25:30.142034057Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","data"],"pid":7,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_xpack => unable to verify the first certificate"}
2020-05-09T08:25:30.150702631Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.151361924Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.153163439Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","savedobjects-service"],"pid":7,"message":"Unable to retrieve version information from Elasticsearch nodes."}
2020-05-09T08:25:30.158642587Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.159443419Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.160227781Z Could not create APM Agent configuration: No Living connections
2020-05-09T08:25:30.161510651Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.162179092Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.162943621Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
2020-05-09T08:25:32.5542814Z {"type":"log","@timestamp":"2020-05-09T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:32.554928902Z {"type":"log","@timestamp":"2020-05-09T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:35.052432592Z {"type":"log","@timestamp":"2020-05-09T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:35.052826311Z {"type":"log","@timestamp":"2020-05-09T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:37.555127441Z {"type":"log","@timestamp":"2020-05-09T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:37.555801219Z {"type":"log","@timestamp":"2020-05-09T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:40.058631235Z {"type":"log","@timestamp":"2020-05-09T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:40.05908746Z {"type":"log","@timestamp":"2020-05-09T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:42.555943297Z {"type":"log","@timestamp":"2020-05-09T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:42.556386251Z {"type":"log","@timestamp":"2020-05-09T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:45.059285085Z {"type":"log","@timestamp":"2020-05-09T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:45.059762989Z {"type":"log","@timestamp":"2020-05-09T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:47.559567891Z {"type":"log","@timestamp":"2020-05-09T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:47.560105532Z {"type":"log","@timestamp":"2020-05-09T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:50.060813521Z {"type":"log","@timestamp":"2020-05-09T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:50.061180134Z {"type":"log","@timestamp":"2020-05-09T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}

And this is yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: <my ip>
    tls:
      certificate:
        secretName: <my secret>
  version: 7.6.2
  nodeSets:
    - name: node
      count: 2
      config:
        node.master: true
        node.ingest: true
        node.data: true
        node.store.allow_mmap: true
      podTemplate:
        metadata:
          labels:
            name: node
          annotations:
            "co.elastic.logs/module": elasticsearch
            "co.elastic.metrics/module": elasticsearch
            "co.elastic.metrics/period": "10s"
            "co.elastic.metrics/hosts": "${data.host}:80"
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 1
                limits:
                  memory: 4Gi
                  cpu: 1
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms2g -Xmx2g"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 200Gi
            storageClassName: standard
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.6.2
  count: 1
  elasticsearchRef:
    name: quickstart
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: <my ip>
    tls:
      certificate:
        secretName: <my secrets>
  podTemplate:
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 2Gi
              cpu: 500m
            limits:
              memory: 2Gi
              cpu: 500m
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms1g -Xmx1g"

And when I connect to Elasticsearch with Chrome, there is no warning with the certificate.
image

Any further guidance on this issue for @Bingu_Shim and I?

Are you including the CA in your custom certificate? It is possible that the CA is in the chrome certificate store already but not in the Elasticsearch image's. You can also try setting logging.verbose: true in the Kibana config to see if there are additional logs.

Yes, I have included the intermediate certificate.

And here is the verbose level logs, there was no additional information for the error.

I guess the problem is Kibana is connecting to the the Elasticsearch using Kubernetes internal domain which is different with my Custom Certificate.

There would be two solution for this problem, i guess

  1. Adding an additional Ingress that handle external domain and certificate
  2. Change Kibana to connect Elasticsearch using external domain, as described HERE
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","plugins","bfetch"],"pid":6,"message":"Initializing plugin"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["info","plugins","bfetch"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: usageCollection"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,cloud"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","legacy-service"],"pid":6,"message":"setting up legacy service"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","root"],"pid":6,"message":"starting root"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","server"],"pid":6,"message":"starting server"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","savedobjects-service"],"pid":6,"message":"Starting SavedObjects service"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: migrations"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nHEAD https://quickstart-es-http.default.svc:9200/.apm-agent-configuration => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_xpack => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
Could not create APM Agent configuration: No Living connections

I guess the problem is Kibana is connecting to the the Elasticsearch using Kubernetes internal domain which is different with my Custom Certificate.

We configure Kibana to skip verifying the host name by default (elasticsearch.ssl.verificationMode: certificate) to enable this use case. Since you are receiving an error it cannot validate the chain, may be worth double checking that you have the whole CA chain provided and it validates on its own (without any pre-installed certificates).

Hello, Anya_Sabo

You are right, it solved.
I misconfigured the certificate.

Thank you very much~!!

@Bingu_Shim what was the misconfiguration on your part? I'm still dealing with the issue. The verbose logging didn't expose any additional errors and I can see from the cert being served in chrome that the intermediate is there too.

@lsnyder,

I have to set following following 3 certificates

  • RootCA.crt, ChainCA1.crt, ChainCA2.crt

But, forgot to add ChainCA2.crt when I create Secret.

@Anya_Sabo I'm noticing slightly different kibana config options between 1.0 and 1.1, could this be the source of my issue?

1.1

ssl:
    certificateAuthorities: /usr/share/kibana/config/elasticsearch-certs/ca.crt
    verificationMode: certificate

1.0

ssl:
    verificationMode: certificate

1.0 does not contain a path to CA. I'm guessing as you said the issue i'm hitting on 1.1 is that my CA is not included in that certificateAuthorities location. My CA and certs are all bundled into the same pem which was used to create my secret using:

kubectl create secret generic <cert-name> --from-file=tls.crt=<name>.pem --from-file=tls.key=<name>.key

Any thoughts or guidance from here? Why would it work in 1.0 and not in 1.1. The cert i'm using is the exact same cert, created in the exact same method.

@lsnyder if you can try adding a ca.crt key to your secret (with the entire CA chain) as described here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html#k8s-setting-up-your-own-certificate
it should populate it correctly. There may have been an inadvertent change between 1.0.1 and 1.1.0 to begin setting that, but I'm not sure off the top of my head and would need to look into it further.

Thanks for reporting and pointing us in the right direction @lsnyder. It looks like your issue in particular was a regression in 1.1.0. We tracked it down in https://github.com/elastic/cloud-on-k8s/issues/3082 and a fix should be available in the next release. In the meantime adding the CA explicitly should resolve it. Sorry about that.

Awesome! Thank you for staying on top of this and helping us out. I can confirm that explicitly supplying the CA when creating the secret worked. We were able to get everything working last night by doing so. :+1:

@lsnyder @Bingu_Shim : It seems, you were able to deploy Elastic and Kibana using ECK and custom certificate. Can one of you please share working yaml and commands:

  • elastic and Kibana yaml
  • create secrete command or yaml file

Need it to deploy it to Azure (AKS).

Thanks in advance.

Hello @nikunjbanker

I did posted sample yaml HERE

and this is command that I used.

kubectl create secret generic <my secret> --from-file=ca.crt=<Root CA File>.crt --from-file=tls.crt=<my crt file>.crt --from-file=tls.key=<my key file>.key