Filebeat handles rotated files issue

Elastic Stack Beats
1

Hi all,
i've installed Filebeat on a AKS cluster with 5 nodes.
I've a stranger behaviour with rotated containers log files.
Docker rotation rule is
{ "live-restore": true, "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "5" } }
When some container logs very often, the rotation is quickly. In this case Filebeat doesn't release the file handler and the result is the disk is full very fast.
I tried to use the close_timeout option, but is not enough.
How i can bypass this issue?

filebeat config map :
`filebeat.inputs:

  • type: log
    paths:
    • /var/log/containers/*.log
      scan_frequency: 10s
      close_timeout: 5h
      symlinks: true
      json.message_key: log
      json.keys_under_root: true
      json.add_error_key: true
      multiline.pattern: '^\s'
      multiline.match: after
      fields_under_root: true
      fields:
      type: kube-logs
      node.hostname: ${NODE_HOSTNAME}
      pod.ip: ${POD_IP}
      tags:
    • k8s-app
      filebeat.config.modules:

    Set to true to enable config reloading

    reload.enabled: true
    output.logstash:
    hosts: logging-logstash:5000
    timeout: 60
    bulk_max_size: 1024
    logging.level: info
    logging.selectors: [logstash]`

Thanks

2

The logs are written faster then logstash accepts them. Normally Filebeat keeps the files open until end of file. You have configured close_timeout: 5h. This is a very big value. Using a much lower value can force filebeat to close files much faster.

3

Thanks,
i reduced the close_timeout to 1h, it's enough os should be less then 1h?
close_timeout force to close all the harvester?

4

How fast do you rotate? Some users set it even to 1min.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.