Filebeat running on Kubernetes connected to Elasticsearch using SSL

[jllorente (Javier Llorente)]

I would like to use filebeat on Kubernetes and connect it to a secured Elasticsearch server, using certificates. I did a search to find how to do this but only found an un-answered related question.
After some test I got it, so I will share my findings to help anybody else.
These are the steps:

1. Follow the original documentation to download the yaml file.

2. As shown on the documentation, update this file with the correct details about the Elasticsearch server: hostname, port, user and password

3. Get a copy of the certificate to be used for the connection. In my example, I am using the "elasticsearch-ca.pem" file created when securing the server.

4. Create a Kubernetes secret using the command:
   kubectl create secret generic elastic-ca --from-file=elasticsearch-ca.pem

5. Edit the filebeat-kubernetes.yaml file as follows:
   At the beginning of the file, update the "output.elasticsearch" section content as follows (only the last 2 lines are added from the original content):

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: "https"
      ssl.certificate_authorities:  ["/etc/filebeat_ssl/elasticsearch-ca.pem"]

6. Under the "volumeMounts" section, add the last 3 lines as follows:

        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: ssl
          mountPath: /etc/filebeat_ssl
          readOnly: true

7. Under the "volumes" section, add the last 3 lines as follows:

      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
      - name: ssl
        secret:
          secretName: elastic-ca

8. You can now deploy the updated yaml file with the command:
   kubectl create -f filebeat-kubernetes.yaml

And that's all! I hope you will find this helpful.