# Filebeat on Kubernetes, with log rotations

**URL:** <https://discuss.elastic.co/t/filebeat-on-kubernetes-with-log-rotations/364389>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 5, 2024, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-with-log-rotations/364389 "2024-08-05T13:29:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dudu\_Edri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dudu_edri/32/124307_2.png) [@Dudu\_Edri](https://discuss.elastic.co/u/Dudu_Edri)\
**Post date:** [August 5, 2024, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-with-log-rotations/364389/1 "2024-08-05T13:29:32Z")

</div>

Hello everyone,

I’m encountering issues with Filebeat in a Kubernetes environment where I don’t have control over the log rotation configuration. Here’s a summary of the setup and the challenges:

1. **Log Setup** :

- **Filebeat Configuration** : Filebeat is configured to look for log files in `/var/log/containers/*.log`, which are symlinks to `/var/log/pods/.../0.log`.
- **Kubernetes Log Rotation** : Kubernetes rotates logs by renaming `0.log` to `0.log.<timestamp>` when the file reaches 10MB. It then gzips the old file and retains up to 3 copies, creating a new `0.log` for continued logging.

1. **Filebeat Configuration Issues** :

- **Disabled Options** :
  - **`close_renamed`** : Disabled, so renamed files are not closed.
  - **`close_removed`** : Disabled, so removed files are not closed.
  - **`clean_removed`** : Disabled as well, as required by Filebeat's documentation if `close_removed` is disabled.

- **Active Options** :
  - **`close_inactive`** and **`clean_inactive`** : Can be set to handle inactive files, but setting it correctly is tricky due to size-based rotation (see the problem below).
  - **`ignore_older`** : Prevents processing of files inactive for longer than the specified period.

- **Problem** : Files that are closed and were renamed/removed will not be reopened as Filebeat looks at the other directory that contains the symlinks. If a file becomes inactive for longer than the `ignore_older` and `close_inactive` periods and then starts logging again, Filebeat will not process the new logs. Without using that configuration (`ignore_older` and `clean_inactive`), frequent rotation increases the risk of inode reuse issues, and on the other hand if a file is mistakenly marked as inactive, it won’t be tracked, leading to potential log loss.

My Questions is, what’s the best approach for configuring Filebeat to handle rotated files given that I can't control the Kubernetes log rotation mechanism, taking into account the problem mentioned above, in order to avoid issues with file closure, inode reuse, and log loss?

Any advice or experiences would be greatly appreciated.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Dudu\_Edri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dudu_edri/32/124307_2.png) [@Dudu\_Edri](https://discuss.elastic.co/u/Dudu_Edri)\
**Post date:** [August 13, 2024, 12:46pm UTC](https://discuss.elastic.co/t/filebeat-on-kubernetes-with-log-rotations/364389/2 "2024-08-13T12:46:30Z")

</div>

Posting here the solution I've come up with.  
Instead of using the `container` input, I changed to the `filestream` input with a `fingerprint` file identity option.  
This is how it looks:

```auto
filebeat.inputs:
- type: filestream
  paths:
	- /var/log/containers/*.log
  file_identity:
	fingerprint: ~
  prospector.scanner:
	check_interval: 10s
	symlinks: true
	fingerprint:
	  enabled: true
	  offset: 0
	  length: 256
  close:
	on_state_change:
	  inactive: 5m
	  renamed: false
	  removed: false
  ignore_older: 24h
  clean_inactive: 25h
  clean_removed: false
  parsers:
  - container:
	  stream: all

```

With that:

- Log files are parsed as container logs with a parser
- We don't have the inode reuse issue, since filebeat now identifies files with a hash of their contents instead of device ID + inode ID
- Files are closed after 5m of inactivity, but will be reopened if logs are written again
- Files older than 24h are ignored, and cleaned from the registry file after 25h in order to prevent it from growing forever