Not setting the elasticsearchRef for setting beat output

Elastic Stack Beats
1

Hello,

We have a k8s cluster dedicated running monitoring software. Elastic Search is installed here using the ECK operator and the CRD's.
On another cluster, we have filebeat running using the deprecated helm charts. So Elastic Search is not running in the same cluster. Using the filebeat config output.elasticsearch we have it configured to send data to Elastic Search running in the external cluster.

We are now trying to move to the newer Helm charts, using the ECK operator and CRDs. We run into problems setting up filebeat with output to our external Elastic Search.
When reading the documentation, I stumbled upon Configuration | Elastic Cloud on Kubernetes [2.9] | Elastic
" Output can be set to any value that is supported by a given Beat. To use it, remove the elasticsearchRef element from the specification and include an appropriate output configuration in the config or configRef elements."

The problem is, this does not work as expected. We have our config as follows:

    config:
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_labels:
            labels:
              group: customer
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      logging.level: error
      output.elasticsearch:
        hosts: ["https://es.domain.tld:443"]
        api_key: ${ELASTICSEARCH_API_KEY}
        pipeline: filebeat-mask

And we did not add any elasticsearchRef. Still, the operator adds a default elasticsearchRef:

elasticsearchRef:
    name: elasticsearch

This results in an error Association backend for elasticsearch is not configured and the DaemonSet is not created by the operator. If I understand the documentation, I did not expect the elasticsearchRef to be added by the operator.

The only way we are able to work around this, is by explicitly setting the ref to an empty value in the spec:

    elasticsearchRef:
        name: ''

But this does not feel right. Are we missing something in our config? Or is this the right way? This is our full spec for the filebeat Helm chart:

  fullname: filebeat
  spec:
    elasticsearchRef:
        name: ''
    type: filebeat
    daemonSet:
      podTemplate:
        spec:
          serviceAccount: elastic-beat-filebeat
          automountServiceAccountToken: true
          containers:
          - name: filebeat
            env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
    secureSettings:
    - secretName: beats-secret
    config:
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_labels:
            labels:
              group: customer
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      logging.level: error
      output.elasticsearch:
        hosts: ["https://es.domain.tld:443"]
        api_key: ${ELASTICSEARCH_API_KEY}
        pipeline: filebeat-mask
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.