Filebeat SSL connection to logstash

[ruchil (Raj Uchil)]

Hi All,
Appreciate any help in configuring SSL connection from Filebeat to logstash on ECK.
Openshift 4.8.x
ECK 2.1.0
ELKF stack 8.1.0

I am using to certs from Elasticsearch-es-http-certs-internal. Created pkcs8.key using the tls.key in Elasticsearch-es-http-certs-internal

oc extract secret/elasticsearch-es-http-certs-internal

openssl pkcs8 -inform PEM -in tls.key -topk8 -nocrypt -outform PEM -out pkcs8/tls.key

oc create secret generic apps-pks-certs --from-file=tls.key=pkcs8/tls.key

logstash.conf

  logstash.conf: |
    input {
      beats {
        port => 5044
        ssl => true
        ssl_certificate_authorities => ["/usr/share/logstash/certs/ca.crt"]
        ssl_certificate => "/usr/share/logstash/certs/tls.crt"
        ssl_key => "/usr/share/logstash/pkcs8/tls.key"
        ssl_verify_mode => "peer"
      }
    }

Filebeat.yaml

          output.logstash:
            hosts: ['logstash.elastic-elk.svc:5044']
            ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
            ssl.certificate: "/etc/filebeat/certs/tls.crt"
            ssl.key: "/etc/filebeat/pkcs8/tls.key"

Logstash error when receviving logs from filebeat

[INFO ] 2022-04-25 16:07:52.146 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 0.0.0.0:5044, remote: 10.131.8.1:49578] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate (caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate)

[michael.morello]

@ruchil

Created pkcs8.key using the tls.key in Elasticsearch-es-http-certs-internal

I would avoid to use the key in elasticsearch-es-http-certs-internal, this key is managed internally by the operator for the purpose of the Elasticsearch HTTPS service. If you want to create a secure connection between Filebeat and Logstash I would create a dedicated cert and key. Back to your example I'm not sure to understand how /etc/filebeat/certs/tls.crt or /usr/share/logstash/certs/tls.crt are created ? And also how it allows Filebeat to trust the name logstash.elastic-elk.svc?

[ruchil (Raj Uchil)]

Hi @michael.morello , I was just being lazy hence used the certs in Elasticsearch-es-http-certs-internal. But now I am using the service-ca operator in openshift to create these certs as per this doc.

Annotated the logstash service as below which creates secret apps-logstash with tls.crt and tls.key

service.alpha.openshift.io/serving-cert-secret-name: apps-logstash

Created configMap ca-bundle for the ca.crt

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: ca-bundle
        namespace: elastic-elk
        annotations:
          service.beta.openshift.io/inject-cabundle: "true"

Mounted the secret and configmap as volumes in both logstash and filebeat pods

    volumeMounts:
          - name: pkcs8-volume
            mountPath: /var/run/secrets/java.io/keystores
          - name: apps-logstash
            mountPath: /var/run/secrets/openshift.io/app-certs
          - name: ca-bundle
            mountPath: /var/run/secrets/openshift.io/ca-bundle
      volumes:
        - name: pkcs8-volume
          emptyDir: {}
        - name: apps-logstash
          secret:
            secretName: apps-logstash
            defaultMode: 420
            optional: true
        - name: ca-bundle
          configMap:
            name: ca-bundle
            defaultMode: 420
            optional: true

As filebeat-logstash SSL requires pkcs8.key, converted the tls.key to pkcs8.key in initcontainers in both pods

      initContainers:
      - name: init-logstash
        image: registry.redhat.io/rh-sso-7/sso75-openshift-rhel8:7.5-23
        command: ['sh', '-c', 'openssl pkcs8 -inform PEM -in /var/run/secrets/openshift.io/app-certs/tls.key -topk8 -nocrypt -outform PEM -out /var/run/secrets/java.io/keystores/pkcs8.key'] 
        volumeMounts:
          - name: apps-logstash
            mountPath: /var/run/secrets/openshift.io/app-certs
          - name: pkcs8-volume
            mountPath: /var/run/secrets/java.io/keystores

Now I am getting a different error

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication (caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication)

logstash config

  apps.conf: |
    input {
      beats {
        port => 5044
        ssl => true
        ssl_certificate => "/var/run/secrets/openshift.io/app-certs/tls.crt"
        ssl_key => "/var/run/secrets/java.io/keystores/pkcs8.key"
        ssl_certificate_authorities => ["/var/run/secrets/openshift.io/ca-bundle/service-ca.crt"]
        ssl_verify_mode => "none"
      }
    }

beats.yml

          output.logstash:
            hosts: ['apps-logstash.elastic-elk.svc:5044']
            ssl.enabled: true
            ssl.certificate:  "/var/run/secrets/openshift.io/app-certs/tls.crt"
            ssl.key: "/var/run/secrets/java.io/keystores/pkcs8.key"
            ssl.certificate_authorities: ["/var/run/secrets/openshift.io/ca-bundle/service-ca.crt"]

Not sure how to proceed further, appreciate any help.