Unable to use Let's Encrypt with ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi every1,
I want use the cert-manager to generate a Let's Encrypt certificate for my ECK. I followed the instructions on Custom HTTP certificate | Elastic Cloud on Kubernetes [1.5] | Elastic and modified the issuer to match the Let's Encrypt ACME:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: my-mail@my-domain.tld
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: quickstart-es-cert
spec:
  dnsNames:
    - my-domain.tld
  issuerRef:
    kind: Issuer
    name: selfsigned-issuer
  secretName: quickstart-es-cert
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.12.1
  nodeSets:
  - name: master-nodes
    count: 1
    config:
      node.roles: ["master"]
      node.store.allow_mmap: false # TODO: Undo for production!
      http.cors.enabled: true
      http.cors.allow-origin: "https://app.elasticvue.com"
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: csi-cinder-high-speed
  - name: data-nodes
    count: 1
    config:
      node.roles: ["data"]
      node.store.allow_mmap: false # TODO: Undo for production!
      http.cors.enabled: true
      http.cors.allow-origin: https://app.elasticvue.com"
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: csi-cinder-high-speed
  http:
    tls:
      certificate:
        secretName: quickstart-es-cert
    service:
      spec:
        type: LoadBalancer

Unfortunately this doesn't work. The pods created are failing with the following error message:

lasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: CertificateParsingException[Empty issuer DN not allowed in X509Certificates];

lasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: CertificateParsingException[Empty issuer DN not allowed in X509Certificates];
Likely root cause: java.security.cert.CertificateParsingException: Empty issuer DN not allowed in X509Certificates
at java.base/sun.security.x509.X509CertInfo.parse(X509CertInfo.java:658)
at java.base/sun.security.x509.X509CertInfo.<init>(X509CertInfo.java:169)
at java.base/sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1791)
at java.base/sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:183)
at java.base/sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:476)
at java.base/sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:361)
at java.base/java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:478)
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:99)
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:91)
at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:52)
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439)
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1224)
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528)
at java.base/java.util.HashMap.forEach(HashMap.java:1425)
at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521)
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526)
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144)
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:462)
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:292)
at org.elasticsearch.node.Node.lambda$new$17(Node.java:567)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.elasticsearch.node.Node.<init>(Node.java:571)
at org.elasticsearch.node.Node.<init>(Node.java:278)
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217)

When I check the certificate issued by the cert-manager it tells me the following:

~$ kubectl describe certs
Name:         quickstart-es-cert
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2021-05-06T08:58:07Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:secretName:
    Manager:      kubectl
    Operation:    Update
    Time:         2021-05-06T08:58:07Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:nextPrivateKeySecretName:
        f:notAfter:
        f:notBefore:
        f:renewalTime:
    Manager:         controller
    Operation:       Update
    Time:            2021-05-06T08:58:08Z
  Resource Version:  8373827242
  Self Link:         /apis/cert-manager.io/v1/namespaces/default/certificates/quickstart-es-cert
  UID:               REMOVED :-)
Spec:
  Dns Names:
    my-domain.tld # REMOVED :-)
  Issuer Ref:
    Kind:       Issuer
    Name:       selfsigned-issuer
  Secret Name:  quickstart-es-cert
Status:
  Conditions:
    Last Transition Time:        2021-05-06T08:58:07Z
    Message:                     Existing issued Secret is not up to date for spec: [spec.dnsNames]
    Observed Generation:         1
    Reason:                      SecretMismatch
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2021-05-06T08:58:08Z
    Message:                     Existing issued Secret is not up to date for spec: [spec.dnsNames]
    Observed Generation:         1
    Reason:                      SecretMismatch
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  quickstart-es-cert-sdv8m
  Not After:                     2021-08-04T08:40:54Z
  Not Before:                    2021-05-06T08:40:54Z
  Renewal Time:                  2021-07-05T08:40:54Z
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    18m   cert-manager  Existing issued Secret is not up to date for spec: [spec.dnsNames]
  Normal  Reused     18m   cert-manager  Reusing private key stored in existing Secret resource "quickstart-es-cert"
  Normal  Requested  18m   cert-manager  Created new CertificateRequest resource "quickstart-es-cert-pkktx"

I'm not sure, but I think that the problem is that the domain isn't pointing to the load balancer atm. The problem is that I'm having a floating IP. So whenever I create a LoadBalancer I'm getting a new IP and have to update the A-Record of my domain. In the meantime the cert-manager tries to issue a certificate but the http01-challenge fails because of the wrong IP. Is my assumption correct or am I miss something in my configuration?

2

Hi @timvol Welcome to the community and thanks for trying ECK.

So I set up my ECK with letsencrypt and a DNS provider (I used GoDaddy example) so I am pretty sure we can get this to work. Yes seems weird how do I create a cert when I don't have the IP

The macro process that I follow is the following.

  • Create cert using certbot the DNS TXT challenge method, this is based on FQDN not IP
  • Install the certs into ECK and Deploy.
  • Get the External IP of your Loadbalacer
  • Go back to your DNS and create an A Record that now points the domain to the IP of the new Load Balancer and it should work. This is the general method I use to create certs when I don't know the IP yet.
  • Access your ECK Kibana / Elasticsearch via FQDN

First create fullchain.pem and privkey.pem using certbot using the TXT challenge method. This method creates a cert based on the Domain Name and the TXT challenge method.

sudo certbot certonly --agree-tos --manual --preferred-challenges dns --server https://acme-v02.api.letsencrypt.org/directory -d "mykibana.mydomain.net"

Create the TXT challenge record in your DNS.

Make sure that it has propagated I use MXTools for that.

Then finish up and you should have your fullchain.pem and privkey.pem

You can install them into your ECK env with the following

kubectl create secret generic my-es-cert --from-file=tls.crt=fullchain.pem --from-file=tls.key=privkey.pem

BTW I did a post with using the ingress controller as well on GKE might be worth a peak.

Hope this helps

3

Hi @stephenb Thank you very much for your help. I was able to solve the "issue" using a separate nginx ingress. So my nginx ingress becomes the LoadBalancer with an official IP address I can issue an certificate for. In the ingress I finally forwarded port 9200 exposed from the http-service created by ECK.

1 Like
4

Nice! Yeah is ingress nice with a static IP.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.