Filebeat multiple path and configs

Hi,

I use filebeat for catching logs from kubernetes.

Actual config is

   - type: container
  paths:
    - /var/log/containers/*.log

I want to use a multiline config for some of my logs (/var/log/containers/celery-*.log), and another multiline config for all other logs (/var/log/containers/ -everything except celery- * logs).
I'd like to keep the wildcard definition instead of listing all logs file so it stays dynamic.

How can I achieve this ?
regards
thomas

The easiest approach is probably using the include_files / exclude_files options. You would specify two container inputs with the same path as you currently have, then in one of them add include_files: ['celery-*.log'] (which will cause it to ignore all files that don't match that pattern) and in the other exclude_files: ['celery-*.log']. Then you can customize the input settings for each case as you like.

1 Like

hi, thanks for answering me.

include_files option does not exists , but idea was good with exclude_files.

I used this syntax to separate config :

filebeat.inputs:
- type: container
  exclude_files: ['celery']
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
- type: container
  paths:
    - /var/log/containers/celery-*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"