Filebeats on ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

While collecting logs from all pods using a filebeat daemonset in an ECK cluster, is there a way to use filebeat modules in this setup?
Most filebeat modules expect a path value typically var.paths. In kubernetes environment, this value is ever-changing. Take, for example, you have a Redis deployment and you want to apply the Redis Filebeat module only to the logs coming from Redis server pods. I am looking for a solution to achieve this.

2

Hi,

If you redis deployments has some specific labels (for example app: "redis") I think that you can use the autodiscover feature and use the condition.contains to match them. Then you should be able to configure the redis module accordingly. For example something along those lines should work:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 7.9.1
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat.autodiscover.providers:
    - node: ${NODE_NAME}
      type: kubernetes
      hints.default_config.enabled: "false"
      templates:
      - condition.contains:
          kubernetes.labels.app: "redis"
        config:
        - module: redis
          log:
            enabled: true
            var.paths:  ["/var/log/containers/*${data.kubernetes.container.id}.log"]
            input:
              type: container
              containers.ids:
                - ${data.kubernetes.container.id}
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers