ECK operator throws "certificate signed by unknown authority"

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi! We are trying to configure an ECK operator (version 1.3.2) that manages an Elasticsearch (6.8.14).

We have configured the Elasticsearch to use our own custom certificates via xpack:

xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/custom-certs/cert.pem
xpack.security.http.ssl.certificate_authorities:
  - /usr/share/elasticsearch/config/custom-certs/ca.pem # our custom ca
  - /usr/share/elasticsearch/config/http-certs/ca.crt # ca already provided
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/custom-certs/key.pem
xpack.security.http.ssl.verification_mode: certificate

The nodes are happily able to talk to each other and our REST clients can also talk to these nodes. However, the Operator itself throws an error when trying to get the cluster settings, which prevents it from making any progress in reconciliation:

{"log.level":"error","@timestamp":"2021-02-26T02:08:35.850Z","log.logger":"controller","message":"Reconciler error","service.version":"1.3.2+cd7cb18e","service.type":"eck","ecs.version":"1.4.0","controller":"elasticsearch-controller","name":"elasticsearch","namespace":"namespace","error":"Put "https://elasticsearch-es-http.namespace.svc:9200/_cluster/settings\": x509: certificate signed by unknown authority","errorCauses":[{"error":"Put "https://elasticsearch-es-http.namespace.svc:9200/_cluster/settings\": x509: certificate signed by unknown authority"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}

I can't seem to find any docs for how we can configure ECK to use our custom certs. Any suggestions or advice? Could we configure Elasticsearch to also accept the certs that ECK is using (where are those controlled?)?

Thanks!

2

The documentation on how to use your own certificates for the HTTP layer of Elasticsearch can be found here TLS certificates | Elastic Cloud on Kubernetes [1.4] | Elastic

Maybe we need to add another link from the Elasticsearch section of the docs to this chapter as it seems hard to find.

What you need to do is to remove all the custom configuration you have right now and instead put the certificate (chain), its private key and if it is not signed by a well known CA also the CA certificate into a secret as outlined in the doc and then reference that secret in your Elasticsearch manifest.

spec:
  http:
    tls:
      certificate:
        secretName: my-cert-secret

ECK will then amend the Elasticsearch configuration just as you did it manually. The difference is that it will be aware of the custom certificates and will also use them when communicating with the cluster.

1 Like
3

Thanks for the answer Peter - it was a little tricky to find and even trickier to apply to our deployment infrastructure, but we got it working eventually.

Just to give some background:

  • Most of our K8s services, including ES clients, are managed by our central infrastructure which mounts custom signed certificates at runtime.
  • Our ES cluster pods are also given this certificate, and we override the REST xpack security settings so that our client is able to properly communicate with the cluster.
  • Meanwhile, we maintain the self signed Operator-managed certs for the transport interface.

The above works fine except that the Operator tries to talk to the cluster via REST, so it needs our custom CA.

We went with your secret approach, but only for the CA (the Cert and Key values are incorrect, but overridden by our ES xpack settings anyway). We can't provide a valid Cert and Key that works across all nodes and the operator.

I think a slightly cleaner way that we would have preferred to provide these certs is via Golang's SystemCertPool environment variables. This way, we could mount real certs at runtime to the pods and the Operator would use them.

Cheers.

1 Like
4

Just for my understanding: your certificate provisioning issues a separate certificate for the HTTP APIs of each Elasticsearch node? You also have just one CA certificate which is used to sign certificates for all applications in you k8s cluster?

5

Yes, both of those statements are correct.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.