Filebeat on Kubernetes, with log rotations

Elastic Stack Beats
1

Hello everyone,

I’m encountering issues with Filebeat in a Kubernetes environment where I don’t have control over the log rotation configuration. Here’s a summary of the setup and the challenges:

  1. Log Setup:
  • Filebeat Configuration: Filebeat is configured to look for log files in /var/log/containers/*.log, which are symlinks to /var/log/pods/.../0.log.
  • Kubernetes Log Rotation: Kubernetes rotates logs by renaming 0.log to 0.log.<timestamp> when the file reaches 10MB. It then gzips the old file and retains up to 3 copies, creating a new 0.log for continued logging.
  1. Filebeat Configuration Issues:
  • Disabled Options:
    • close_renamed: Disabled, so renamed files are not closed.
    • close_removed: Disabled, so removed files are not closed.
    • clean_removed: Disabled as well, as required by Filebeat's documentation if close_removed is disabled.
  • Active Options:
    • close_inactive and clean_inactive: Can be set to handle inactive files, but setting it correctly is tricky due to size-based rotation (see the problem below).
    • ignore_older: Prevents processing of files inactive for longer than the specified period.
  • Problem: Files that are closed and were renamed/removed will not be reopened as Filebeat looks at the other directory that contains the symlinks. If a file becomes inactive for longer than the ignore_older and close_inactive periods and then starts logging again, Filebeat will not process the new logs. Without using that configuration (ignore_older and clean_inactive), frequent rotation increases the risk of inode reuse issues, and on the other hand if a file is mistakenly marked as inactive, it won’t be tracked, leading to potential log loss.

My Questions is, what’s the best approach for configuring Filebeat to handle rotated files given that I can't control the Kubernetes log rotation mechanism, taking into account the problem mentioned above, in order to avoid issues with file closure, inode reuse, and log loss?

Any advice or experiences would be greatly appreciated.

Thanks in advance!

2

Posting here the solution I've come up with.
Instead of using the container input, I changed to the filestream input with a fingerprint file identity option.
This is how it looks:

filebeat.inputs:
- type: filestream
  paths:
	- /var/log/containers/*.log
  file_identity:
	fingerprint: ~
  prospector.scanner:
	check_interval: 10s
	symlinks: true
	fingerprint:
	  enabled: true
	  offset: 0
	  length: 256
  close:
	on_state_change:
	  inactive: 5m
	  renamed: false
	  removed: false
  ignore_older: 24h
  clean_inactive: 25h
  clean_removed: false
  parsers:
  - container:
	  stream: all

With that:

  • Log files are parsed as container logs with a parser
  • We don't have the inode reuse issue, since filebeat now identifies files with a hash of their contents instead of device ID + inode ID
  • Files are closed after 5m of inactivity, but will be reopened if logs are written again
  • Files older than 24h are ignored, and cleaned from the registry file after 25h in order to prevent it from growing forever