Can filebeat autodiscover collect logs from k8s emptydir volume?

Elastic Stack Beats
1

Here is my filebeat config yaml, downloaded from Run Filebeat on Kubernetes | Filebeat Reference [7.15] | Elastic

I can't get log when the templates.config.type is log (yaml below), But when I set this to container I can get the filebeat output ( it's parsing my logs mounted from pod emptydir, reporting unable to parse nginx access log as json, which means it can retrive my nginx log ), but when I change type to log, nothing output.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         hints.enabled: true
         hints.default_config.enabled: false
         templates:
         - config:
            - type: log 
              paths:
                - /var/lib/kubelet/pods/${data.kubernetes.pod.uid}/volumes/kubernetes.io~empty-dir/logs/*.log
                - /var/lib/kubelet/pods/${data.kubernetes.pod.uid}/volumes/kubernetes.io~empty-dir/logs/*.txt

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---

I 've added annotation to my pod

annotations:
        co.elastic.logs/enabled: "true"
        co.elastic.logs/multiline.pattern: '^\['
        co.elastic.logs/multiline.negate: true
        co.elastic.logs/multiline.match: after
        co.elastic.logs.sidecar/exclude_lines: '^DBG'
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.