Filebeat handler open rotated log files

On our kuberntes cluster we rotate the logs using the dockerd option "--log-opt max-size and --log-opt max-file". When the log are rotated filebeat still hold the rotated log file. We keep only 3 rotated log files, this will result in fielbeat still referencing deleted files.

The issue is similar to Filebeat Holds open rotated log files. But i did not see any workaround or solution.

I see option "close_rename" but not sure how safe is to enable this option. Or is there any other solution? Im using filebeat version 6.7 Please let me know.

To add to it, it looks like its not closing the file "docker_id.log.1" even after the close_inactive time of 5m

Filebeat configs are safe.

I am using the closing options (provided below) and do see file closing due to inactivity

File is inactive: /home/xyz/log//stderr.txt. Closing because close_inactive of 5m0s reached.

Did you try close_removed: true for stopping harvesters when old files are deleted ?

 ### Harvester closing options

  # Close inactive closes the file handler after the predefined period.
  # The period starts when the last line of the file was, not the file ModTime.
  # Time strings like 2h (2 hours), 5m (5 minutes) can be used.
  close_inactive: 5m

  # Close renamed closes a file handler when the file is renamed or rotated.
  # Note: Potential data loss. Make sure to read and understand the docs for this option.
  close_renamed: false

  # When enabling this option, a file handler is closed immediately in case a file can't be found
  # any more. In case the file shows up again later, harvesting will continue at the last known position
  # after scan_frequency.
  close_removed: true

  # Closes the file handler as soon as the harvesters reaches the end of the file.
  # By default this option is disabled.
  # Note: Potential data loss. Make sure to read and understand the docs for this option.
  close_eof: false

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.