Filebeat doesn't log anythign with kubernetes autodiscover

[lassizci (Lassi Pölönen)]

Hi, I've been trying to set up kubernetes logging with filebeat from outside our kubernetes cluster. Here is my config:

filebeat:
  autodiscover:
    providers:
      - type: kubernetes
        kube_config: /etc/kubernetes/kubelet-kubeconfig.yml
        in_cluster: false
        templates:
          - condition:
              regexp:
                kubernetes.namespace: ".*"
            config:
              - type: docker
                include_annotations: true
                containers.ids:
                  - "${data.kubernetes.container.id}"
logging:
  files:
    keepfiles: 7
    name: filebeat.log
    path: /var/log/filebeat
    permissions: '0644'
    rotateeverybytes: 104857600
  level: debug
  to_files: true
output:
  file:
    path: /tmp/filebeat

Filebeat doesn't seem to even create the output file.

Logs here (due to body size limit..): https://pastebin.com/ry0bvA6f

To me it looks like it does find at least some pods, even though only for kube-system namespace while I have others running there too. But no output from those produced either.

I've tried previously without the autodiscover:

filebeat:
  prospectors:
  - fields:
      kubeenv: dev1
      type: kubelog
    fields_under_root: true
    json:
      keys_under_root: true
      message_key: log
    paths:
    - /var/lib/docker/containers/*/*.log
    processors:
    - add_kubernetes_metadata:
        in_cluster: false
        kube_config: /etc/kubernetes/kubelet-kubeconfig.yml
    type: log

and that seems to be working though, but there's an issue with that: https://github.com/elastic/beats/issues/5377

Any ideas what possibly could be wrong?

[lassizci (Lassi Pölönen)]

Turned out to be an issue with machine-id. Added host to provider config and started to see logs, so the detection went wrong because provisioning hadn't generated a unique one. Kubelet credentials don't apparently have watch permission by default, but didn't seem to be a part of the issue. Better practice anyway is to create a separate read-only credentials for filebeat.