Issue with Filebeat Kubernetes Configuration for Ingress Logs

[Ankur_Mahajan]

I'm attempting to use Filebeat on Kubernetes to ship my ingress logs and take advantage of the Nginx module dashboard. I've followed the configurations provided in the documentation, but I'm encountering an unusual issue with the autodiscover template condition. After verifying the data in the Filebeat logs, it appears that Filebeat is sending all container logs to the Filebeat index, which doesn't seem to be the expected behaviour.

I've reviewed various blogs and noticed that similar configurations are recommended across the board. However, in my case, all container logs are being sent, whereas I specifically want to collect syslog and Nginx logs. For your reference, please review the filebeat-module.yaml configuration file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-module-config
  namespace: elk
  labels:
    k8s-app: filebeat-module
data:
  filebeat.yml: |-
    logging.level: debug
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
            - condition:
                  equals:
                    kubernetes.container.name: "controller"
              config:
              - module: nginx
                access:
                  enabled: true
                  input:
                    type: container
                    paths:
                    - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log

    filebeat.modules:
    - module: system
      syslog:
        enabled: true
      auth:
        enabled: true

    processors:
    - add_fields:
        target: ''
        fields:
          env: ${INSTANCE}

    setup.dashboards:
      enabled: true

    output.elasticsearch:
      hosts: ["${ES_HOST_URL}"]
      username: ${ES_USER}
      password: ${ES_PASSWORD}
      ssl.verification_mode: none

    setup.kibana:
      host: ${KIBANA_HOST_URL}
      ssl.verification_mode: none
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat-module
  namespace: elk
  labels:
    k8s-app: filebeat-module
spec:
  selector:
    matchLabels:
      k8s-app: filebeat-module
  template:
    metadata:
      labels:
        k8s-app: filebeat-module
    spec:
      serviceAccountName: elk-beats
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat-module
        image: docker.elastic.co/beats/filebeat:7.9.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e"
        ]
        env:
        - name: STACK_NAMESPACE
          valueFrom:
            configMapKeyRef:
              name: elk-configmap
              key: stack_namespace
        - name: INSTANCE
          valueFrom:
            configMapKeyRef:
              name: elk-configmap
              key: environment
        - name: ES_HOST_URL
          valueFrom:
            configMapKeyRef:
              name: elk-configmap
              key: esUrl
        - name: KIBANA_HOST_URL
          valueFrom:
            configMapKeyRef:
              name: elk-configmap
              key: kibanaUrl
        - name: ES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elk-secrets
              key: password
        - name: ES_USER
          valueFrom:
            secretKeyRef:
              name: elk-secrets
              key: username
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-module-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      - name: data
        hostPath:
          path: /var/lib/filebeat-module-data
          type: DirectoryOrCreate

[Ankur_Mahajan]

Adding drop_event resolved one part of the issue, but it introduced another problem – now, syslogs are also being dropped. Do you have any suggestions for addressing this issue?

    - drop_event:
       when:
         not:
           equals:
              kubernetes.container.name: "controller"

I have tried multiple condition patterns like the below but no luck.

    - drop_event:
        when:
          or:
            - and:
                - equals:
                    kubernetes.container.name: "controller"
                - not:
                    equals:
                      event.module: "system"
            - and:
                - not:
                    equals:
                      kubernetes.container.name: "controller"
                - equals:
                    event.module: "system"
            - and:
                - not:
                    exists:
                      kubernetes.container.name
                - equals:
                    event.module: "system"
            - and:
                - equals:
                    kubernetes.container.name: "controller"
                - not:
                    exists:
                      event.module