Filebeat how to autodiscover kubernetes service and fetch pod log?

[chanjarster (Daniel Qian)]

I've deploy filebeat on kubernetes cluster following the doc. And set resource: service in the config file:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          resource: service
          node: ${NODE_NAME}
          include_annotations: []
          include_labels: []
          hints.enabled: true
          hints.default_config:
            enabled: false
            type: container
            paths:
              - /var/log/containers/*-${data.kubernetes.container.id}.log
    output.console:
      pretty: true

And then I add co.elastic.logs/enabled: "true" on a Service.

Filebeats discovered that service but print a error:

2020-06-22T03:27:25.307Z ERROR [autodiscover] autodiscover/autodiscover.go:210 Auto discover config check failed for config '{
"docker-json": {
"cri_flags": true,
"format": "auto",
"partial": true,
"stream": "all"
},
"symlinks": true,
"type": "container"
}', won't start runner: each input must have at least one path defined

So, did I missing something?

[chanjarster (Daniel Qian)]

Solved. I use config template solve this.

[rhallier]

Hi Daniel,
I would be very interested if you could post the solution, I've got the same pb
Regards
R.

[chanjarster (Daniel Qian)]

Something like this, but you cannot specify the log file name, because kubernetes log file name is {pod.name}-{pod.namespace}-{container.name}-{container.id}.log

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.image: "redis"
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log