Filebeat and Kubernetes: excluding log files

Elastic Stack Beats
12

Hi @Sven_Woltmann! Thanks a lot for your messages, I finally got it working! :tada:

You're right, indentation was off, and also I didn't have the symlinks: true option on. Here's my final config, and I can confirm it works with filebeat:6.1.3:

filebeat.prospectors:
  - type: log
    paths:
      - "/var/log/containers/*.log"
    exclude_files:
      - filebeat-.*\.log
      - logstash-.*\.log
    json.message_key: log
    json.add_error_key: true
    json.keys_under_root: true
    symlinks: true
    tail_files: true

processors:
  - add_kubernetes_metadata:
      in_cluster: true
      default_matchers.enabled: false
      matchers:
      - logs_path:
          logs_path: /var/log/containers/

filebeat.shutdown_timeout: 5s

output.logstash:
  hosts: ["logstash:5044"]

Also worth mentioning that I'm mounting:

  • /var/lib/docker/containers
  • /var/log/pods
  • /var/log/containers

All three in readOnly mode.

Thanks a lot @exekias and @Sven_Woltmann for your help :slight_smile:

Jeremie

4 Likes