Filebeat not logging hints.enabled container logs

[dev4elasticapm (Dev)]

Hi, I am using ECK, I have configured filebeat with two options

1. hints.enabled: true, which looks for all the containers with co.elastic.logs/enabled: "true"
2. Checks the container containing name ingress.

#2 is working fine for me but I can't figure out how to get #1 working. Below is my filebeat.yaml file.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: search
spec:
  type: filebeat
  version: 7.12.1
  elasticsearchRef:
    name: elastic-search
  kibanaRef:
    name: kibana-web
  config:
    filebeat.autodiscover.providers:
    - node: ${NODE_NAME}
      type: kubernetes
      hints.enabled: true
      #add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: "false"
    - node: ${NODE_NAME}
      type: kubernetes
      #add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: "false"
      templates:
      - condition:
          contains: 
            kubernetes.pod.name: ingress
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          exclude_lines: ["^\\s+[\\-`('.|_]"]
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        #hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              memory: 200Mi
              cpu: 0.2
            limits:
              memory: 300Mi
              cpu: 0.4
              
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

This is how my spring boot app deployment is configured

spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
      annotations:
        co.elastic.logs/enabled: "true" # enable the logs collection for filebeat
        co.elastic.logs/multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'
        co.elastic.logs/multiline.negate: true
        co.elastic.logs/multiline.match: after

I can confirm that the spring app is logging the error logs but it's not being send to ECK Discover filebeat stream.I also confirmed that the Filebeat is running in both of the nodes where spring app is installled.

I think I'm missing a simple configuration in the file. I have tried adding (but didn't work!)

templates:
      - condition:
          contains: 
            co.elastic.logs/enabled: "true"
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          exclude_lines: ["^\\s+[\\-`('.|_]"]

right after

- node: ${NODE_NAME}
      type: kubernetes
      hints.enabled: true
      #add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: "false"

[hendry.lim (Hendry)]

Can you try the following Autodiscover config?

filebeat.autodiscover.providers:
  - type: kubernetes
    node: ${NODE_NAME}
    hints.enabled: true
    hints.default_config.enabled: false
    templates:
      - condition:
          contains: 
            kubernetes.pod.name: ingress
        config:
          - type: container
            paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
            exclude_lines: ["^\\s+[\\-`('.|_]"]

[dev4elasticapm (Dev)]

Hi @hendry.lim Thank you for sharing this. I tried this config but it didn't work. I got the ingress logs but nothing from my web application [spring boot java app]

This is how my filebeat.yaml file looks right now [based on your suggested changes]

spec:
  type: filebeat
  version: 7.12.1
  elasticsearchRef:
    name: elastic-search
  kibanaRef:
    name: kibana-web
  config:
    filebeat.autodiscover.providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      #add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: false
      templates:
      - condition:
          contains: 
            kubernetes.pod.name: ingress
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          exclude_lines: ["^\\s+[\\-`('.|_]"]

[dev4elasticapm (Dev)]

Alright! I finally got the configuration working Below is fully functional filebeat.yaml file. It does following:

1. Checks if co.elastic.logs/enabled: "true" annotation is enabled in any of the container, if it does, then the logs are sent to elasticsearch.

2. Checks if any of the pod name contains nginx. If it does, then the logs are sent to elasticsearch.

Working filebeat.yaml file:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: search
spec:
  type: filebeat
  version: 7.12.1
  elasticsearchRef:
    name: elastic-search
  kibanaRef:
    name: kibana-web
  config:
    filebeat.autodiscover.providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints: 
        enabled: true
        #add_resource_metadata.namespace.enabled: true
        default_config: 
          enabled: false
          type: container
          paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
      templates:
      - condition:
          contains: 
            kubernetes.pod.name: ingress
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        #hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              memory: 200Mi
              cpu: 0.2
            limits:
              memory: 300Mi
              cpu: 0.4
              
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

The key is where you specify the annotation in your application's yaml file. In my case [java app], if I specify the annotation at the top level (i.e. deployment yaml's metadata.annotations level, the logs are not sent. When I set the annotations for the pod (i.e. at spec.template.metadata.annotations), the logs started coming in.