[Filebeat] K8s OOM kill causes file corruption

Hi,

I have the following issue and think it's a bug but wanted to post here before filing a Github issue:

A filebeat container gets killed/restarted because of memory limits within K8s. After the container restart the following log appears and no events are sent from this filebeat pod anymore:

{"level":"warn","timestamp":"2020-10-20T18:21:56.116Z","caller":"memlog/store.go:130","message":"Incomplete or corrupted log file in /usr/share/filebeat/data/xyz/filebeat. Continue with last known complete and consistent state. Reason: unexpected EOF"}

A pod restart resolves the issue though.

Filebeat version 7.9.2 with Logstash output. For log collection I use the Kubernetes autodiscover feature.

Regards
Bernd

Hmmm, as indicated in the log line this is a warning and it shouldn't affect the operation of Filebeat. Would you mind reproducing this issue but with Filebeat logging level set to debug (by setting logging.level: debug in the Filebeat configuration)? And then please post the logs from about 15 seconds before the warning log line through 15 seconds after.

Thanks,

Shaunak

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.