Communicating with ECK using your own certs results in a certificate error

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hey,

I'm following the tutorial teaching how to set your own certificates.
I installed the cert manager who created the quickstart-es-cert secret for me.

$ k describe secret quickstart-es-cert
Name:         quickstart-es-cert
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: quickstart-es-http,quickstart-es-http.default.svc,quickstart-es-http.default.svc.cluster.local
              cert-manager.io/certificate-name: quickstart-es-cert
              cert-manager.io/common-name:
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: selfsigned-issuer
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
tls.crt:  1229 bytes
tls.key:  1675 bytes
ca.crt:   1229 bytes

But when I try to curl ES I get the following error:

$ curl --cacert tls.crt -u elastic:$PW https://$IP:9200/
curl: (60) Certificate type not approved for application.

My cluster's config is the following:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.6.1
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
  http:
    service:
      spec:
        type: ClusterIP
    tls:
      selfSignedCertificate:
        subjectAltNames:
        - ip: 10.233.27.202
      certificate:
        secretName: quickstart-es-cert

Using ECK v1.0
Env: On premise

Any help would be apreciated :wink:
Thx

2

There are a few issues here:

  • In the http section, tls.selfSignedCertificate and tls.certificate are mutually exclusive fields. You should only have one or the other. If you want to use cert-manager to issue the certificate, then only the tls.certificate field should be set.
  • You have set the service type to ClusterIP -- which is the default anyway. It only makes pods accessible inside the Kubernetes cluster. The IP address you are trying to issue the certificate to is a private, internal IP address that cannot be accessed from outside.
  • The documentation page you linked to omits a few minor details that might not be obvious at first. Apologies for that. We'll review it for the next release.

If you are trying to access Elasticsearch from inside the cluster, you don't need to issue the certificate for an IP address. Just use the internal DNS name of the service as described in https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-custom-http-certificate.html#k8s_custom_self_signed_certificate_using_cert_manager

If you are trying to access Elasticsearch from outside the Kubernetes cluster, then the service type must be set to LoadBalancer or NodePort (see https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types). As you're using an on-premise Kubernetes deployment, I am not sure how it will handle these service types. You should check with your system administrator to understand how the cluster is configured to expose services externally.

The following snippets illustrates how one would normally use cert-manager to expose an Elasticsearch deployment. Some details may vary depending on how your Kubernetes cluster is configured (for instance, not all providers support loadBalancerIP).

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: quickstart-es-cert
spec:
  secretName: quickstart-es-cert
  dnsNames:
    - "quickstart-es-http.default.svc.cluster.local"
  ipAddresses:
    - "210.0.0.1"
  issuerRef:
    name: selfsigning-issuer
    kind: ClusterIssuer
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.6.1
  http:
    service:
      spec:
        type: LoadBalancer
        loadBalancerIP: 210.0.0.1
    tls:
      certificate:
        secretName: quickstart-es-cert
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

Hope that helps.

3

Thanks for your answer and for the time you invested.

I used the yaml you provided and updated it to use NodePort.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: quickstart-es-cert
spec:
  secretName: quickstart-es-cert
  dnsNames:
    - "quickstart-es-http.default.svc.cluster.local"
  ipAddresses:
    - "10.10.5.7"
  issuerRef:
    name: selfsigning-issuer
    kind: ClusterIssuer
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.6.1
  http:
    service:
      spec:
        type: NodePort
        ports:
          - name: https
            nodePort: 31111
            port: 9200
            protocol: TCP
            targetPort: 9200
    tls:
      certificate:
        secretName: quickstart-es-cert
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

Unfortunatly this leads to a CrashLoopbackOff of the elastic container withe the following stacktrace:

ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to create trust manager]; nested: ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: CertificateException[failed to parse any certificates from [/usr/share/elasticsearch/config/http-certs/tls.crt]];
Likely root cause: java.security.cert.CertificateException: failed to parse any certificates from [/usr/share/elasticsearch/config/http-certs/tls.crt]

The secret has been created but tls.crt seems to be 0 bytes long:

 k describe secret quickstart-es-cert
Name:         quickstart-es-cert
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: quickstart-es-cert
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: selfsigning-issuer

Type:  kubernetes.io/tls

Data
====
ca.crt:   0 bytes
tls.crt:  0 bytes
tls.key:  1679 bytes

The yaml I used to create the certs until now was the one from the documentation. In the end it wasn't working but at least the certs were there. Should I somehow merge the yaml you provided with the one from the tutorial ?

4

It looks like an issue with cert-manager in your cluster. The certificate should not be empty. Try deleting and re-creating the certificate to see if the problem persists. It might be worth checking the cert-manager logs as well to see if there are any issues reported there.

The cert-manager Certificate manifest in the documentation is correct and you should be able to use it as-is. I was simply illustrating how to add an IP address to the generated certificate as that's what you were trying to do in the original question.

5

I got it working...

I created the cert using the openssl command given in the documentation

$ openssl req -x509 -sha256 -nodes -newkey rsa:4096 -days 365 -subj "/CN=quickstart-es-http" -addext "subjectAltName=DNS:quickstart-es-http.default.svc" -keyout tls.key -out tls.crt
$ kubectl create secret generic quickstart-es-cert --from-file=ca.crt=tls.crt --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key

For the curl to work in https you must add the value of the subjectAltName in the /etc/hosts. If you don't you'll get some strange NSS errors.

So simply add this line to your /etc/hosts

<eck_svc_ip> quickstart-es-http.default.svc

My elastic config is now the following:

...
  http:
    service:
      spec:
        type: ClusterIP
        clusterIP: 10.233.35.89       #Hardcoding value of clusterIP because why not
    tls:
      certificate:
        secretName: quickstart-es-cert     #Secret with new certs

You can now curl your endpoint using the --cacert of curl.

curl https://<subjectAltName>:9200 --cacert ./tls.crt
6

Glad you got it working. Adding the IP address to /etc/hosts is strictly not necessary and a potential security issue if you are using a shared cluster with untrusted users. For testing purposes, you can invoke curl with the --resolve flag instead.

curl --resolve "$DOMAIN:$PORT:$IP_ADDRESS" --cacert ./tls.crt "https://$DOMAIN:$PORT/_cat/health"
7

Thanks for pointing this out.
Setting the value of the service was only convenient for testing purposes . :wink: