Quickstart Example (v0.8) SSL error

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi, I've followed the guide https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html and deployed the items described, everything seems to have been correctly created.

However when the Elasticsearch logs are inspected there is a problem connecting to it which looks like it relates to an SSL certificate. Any advice on how to resolve this?

The relevant extract from the Elasticsearch log is as follows:

{"type": "server", "timestamp": "2019-06-03T14:25:47,157+0000", "level": "WARN", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "quickstart", "node.name": "quickstart-es-p27jq468fh", "cluster.uuid": "zu07afsqR4Su51dScM6MYw", "node.id": "vmGa6_pnRaqjagdW3v4GeA", "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/10.244.0.100:45094}" ,
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate"

The filebeat log is shown below:
2019-06-03T14:27:28.643Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30310,"time":{"ms":6}},"total":{"ticks":153690,"time":{"ms":47},"value":153690},"user":{"ticks":123380,"time":{"ms":41}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"bbd1dd2f-d815-4003-829b-182cc3d8ce3f","uptime":{"ms":268890029}},"memstats":{"gc_next":46221424,"memory_alloc":23183016,"memory_total":3499909608}},"filebeat":{"harvester":{"open_files":9,"running":21}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":4117,"retry":50}}},"registrar":{"states":{"current":26}},"system":{"load":{"1":0.23,"15":0.03,"5":0.07,"norm":{"1":0.115,"15":0.015,"5":0.035}}}}}}

2019-06-03T14:27:58.037Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://quickstart-es.default.svc.cluster.local:9200)): Get https://quickstart-es.default.svc.cluster.local:9200: x509: certificate signed by unknown authority

2019-06-03T14:27:58.037Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://quickstart-es.default.svc.cluster.local:9200)) with 5947 reconnect attempt(s)

2

We are using a self-signed CA internally (we will change this to allow users to specify their own certificates in the near future).

For the time being you need to make any client your are using aware of the custom CA certificate. It looks like you are trying to setup filebeat in your example. So a filebeat configuration that does that could look like this:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOSTS}
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
  ssl.certificate_authorities:
    - /etc/certificate/ca.pem

Where the path /etc/certificate/ca.pem corresponds to a volume mount of the CA certificate the operator uses. The operator stores the CA certificate in a secret that is called $CLUSTERNAME-ca which contains the ca.pem file. We are in the process of improving documentation around this.

3

Hi Peter,

Thanks for your response.

The filebeat configuration is taken from the guide found at https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html with the namespace changed to default.

As suggested, I've added the ssl.certificate_authorities to filebeat-kubernetes.yaml, and re-applied it.

Unfortunately the filebeat daemon-set and pods won't start and are giving the following error:

2019-06-04T10:40:54.895Z	INFO	instance/beat.go:280	Setup Beat: filebeat; Version: 7.0.0
2019-06-04T10:40:54.895Z	INFO	[index-management]	idxmgmt/std.go:165	Set output.elasticsearch.index to 'filebeat-7.0.0' as ILM is enabled.
2019-06-04T10:40:54.897Z	ERROR	tlscommon/tls.go:145	Failed reading CA certificate: open /etc/certificate/ca.pem: no such file or directory
2019-06-04T10:40:54.897Z	INFO	instance/beat.go:361	filebeat stopped.
4

You have to mount the secret containing the CA certificate into the DaemonSet.

The operator stores the CA certificate in a secret that is called $CLUSTERNAME-ca which contains the ca.pem file

You have to reference that in a volume mount, maybe that was not clear from my previous answer. Something along the lines of:

apiVersion: extensions/v1beta1
kind: DaemonSet
...snip...
spec:
  template:
    ...snip...   
    spec:
      containers:
      - name: filebeat-dynamic
        image: docker.elastic.co/beats/filebeat:7.0.1
       ...snip...
        volumeMounts:
        - ... snip...
        - name: certificate
          mountPath: /etc/certificate/ca.pem
          readOnly: true
          subPath: ca.pem
      volumes:
      - ... snip...
      - name: certificate
        secret:
          secretName: $YOURCLUSTER-ca
1 Like
5

Hi Peter,

Thanks ever so much, that worked like a charm. In my case, and to help others following the same examples, the secret $CLUSTERNAME-ca was quickstart-es-ca

Thanks again for a quick and accurate response.

Kind regards

Karl

6

I encountered the same error, but did not find the secret of $clustername ca. I used https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-quickstart.html

7

We have changed the naming scheme for the certificate secrets in the time since the original answer was written. Look for $CLUSTER_NAME-es-http-certs-public instead now.

The correct names are documented for each version here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html#k8s-tls-certificates

8

Your support is very quick. At present, my problem has been solved.
Thanks for your response.